Getting Started

Python Jupyter notebooks are an excellent way to experiment with data science and visualization. Using the higlass-jupyter extension, you can use HiGlass directly from within a Jupyter notebook.

Installation

To use higlass within a Jupyter notebook you need to install a few packages and enable the jupyter extension:

pip install jupyter higlass-python

jupyter nbextension install --py --sys-prefix --symlink higlass
jupyter nbextension enable --py --sys-prefix higlass

If you use JupyterLab you also have to run

jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install higlass-jupyter

Uninstalling

jupyter nbextension uninstall --py --sys-prefix higlass

Simplest Use Case

The simplest way to instantiate a HiGlass instance to create a display object with one view:

import higlass
from higlass.client import Track, View

display, server, viewconf = higlass.display([View([Track('top-axis')])])
display

If brevity is of importance, the constructor for View can be omitted and a view will automatically be created from the list of Tracks: higlass.display([[Track('top-axis')]]). This, however, precludes the use of parameters with the view or for linking views using syncs. It also always uses the default position for a given track type.

Remote Jupyter Notebook

If your Jupyter notebook is running on a remote server (at e.g. REMOTE_IP), you’ll need to make sure that you have an extra port (let’s call it HG_PORT) open on your firewall that HiGlass can use to communicate with its server. That port will then need to be passed to the server display command:

higlass.display(
    ...,
    server_port=HG_PORT,
    host=REMOTE_IP,
    fuse=False
)

The fuse=False option is often necessary if there is no support for FUSE. FUSE is only necessary for loading remote http datasets which are not hosted on a HiGlass server.

View extent

The extent of a view can be set using the initialXDomain parameter:

view1 = View([
    Track(type='top-axis'),
], initialXDomain=[0,1e7])

Search box

Views can have a search box which shows the current genomic position and lets users search for genes. In order for the current position to be shown, we need to pass in a chromsizes track. For gene search to be enabled, we have to pass in a gene annotations track:

chromosomes = Track(tilesetUid='N12wVGG9SPiTkk03yUayUw',
           server='https://higlass.io/api/v1',
           type='horizontal-chromosome-labels')
genes = Track(tilesetUid='OHJakQICQD6gTD7skx4EWA',
           server='https://higlass.io/api/v1',
           type='horizontal-gene-annotations')

(d,s,v) = higlass.display([
    View(
            [chromosomes, genes],
            chrominfo=chromosomes,
            geneinfo=genes,
        )
])
d

Track Types

A list of available track types can be found in the documentation for HiGlass. Based on the data type, we can sometimes provide a recommended track type as well as a recommended position.

import higlass.client as hgc
track_type, position = hgc.datatype_to_tracktype(datatype)

Color Maps

Certain quantative tracks such as the heatmap can vary their colormap. Color maps can be passed in directly as arrays of color values:

Track('heatmap', tileset, colorRange=['white', 'black'])

Or created from a matplotlib colormap (reversed=True reverses the color order in the heatmap):

from higlass.utils import hg_cmap
Track('heatmap', tileset, colorRange=hg_cmap('jet', reverse=True))

A list of available matplotlib color maps can be found in the matplotlib docs.

Combining Tracks

Tracks can be combined by overlaying them on top of each other or by performing operations with them.

Overlaying tracks

Two tracks can be overlayed by using the + operator:

view=View([Track('top-axis') +
       Track('horizontal-bar',
            server='//higlass.io/api/v1',
            tilesetUid='F2vbUeqhS86XkxuO1j2rPA')
      ], initialXDomain=[0,1e9])

Another way to express this is to pass in a list of tracks as if it were a single track:

view=View([[Track('top-axis'),
       Track('horizontal-bar',
            server='//higlass.io/api/v1',
            tilesetUid='F2vbUeqhS86XkxuO1j2rPA')
      ]], initialXDomain=[0,1e9])

Multiple Views

Multiple views can be instantiated much like single views. They are positioned a on grid that is 12 units wide and an arbitrary number of units high. To create two side by side views, set both to be 6 units wide and one on the right to be at x position 6:

import higlass
from higlass.client import Track, View

view1 = View([Track(type='top-axis')], x=0, width=6)
view2 = View([Track(type='top-axis')], x=6, width=6)

display, server, viewconf = higlass.display([view1, view2])
display

Synchronization

Views and track can be synchronized by location, zoom level and values scales.

Zoom and Location locks

Location locks ensure that when one view is panned, all synchronized views pan with it. Zoom locks do the same with zoom level. Both can be instantiated by passing lists of views to lock to higlass.display. Each set of locked views will scroll or zoom (or both) together:

display, server, viewconf = higlass.display(
  [view1, view2],
  location_syncs=[[view1, view2]],
  zoom_syncs=[[view1, view2]])

Viewport Projection

Viewport projections can be instantiated like other tracks. It is created with a reference to the view we wish to track and combined with another track where it will be overlayed.

from higlass.client import ViewportProjection

view1 = View([
    Track(type='top-axis'),
], initialXDomain=[0,1e7])

projection = ViewportProjection(view1)

view2 = View([
    Track(type='top-axis') + projection,
], initialXDomain=[0,2e7])

Note that ViewportProjection tracks always need to be paired with other non- ViewportProjection tracks. Multiple ViewportProjection tracks can, however, be combined, as long as they are associated with regular tracks.

Combined tracks can also be created by passing a list of tracks as if it were a track itself to a View.

view2 = View([
  [ Track(type='top-axis'), projection ]
], initialXDomain=[0,2e7])

Dataset Arithmetic

HiGlass supports client-side division between quantitative datasets. This makes it possible to quickly compare two datasets by visualizing their ratio as computed on loaded tiles rather than the entire dataset:

t1 = Track(**track_def)
t2 = Track(**{ **track_def, "tileset_uuid": "QvdMEvccQuOxKTEjrVL3wA" })
t3 = t1 / t2

They can also be created using a constructor:

from higlass.client import DividedTrack

t3 = DividedTrack(t1, t2)

The full example is here:

from higlass.utils import hg_cmap

track_def = {
    "track_type": 'heatmap',
    "position": 'center',
    "tileset_uuid": 'CQMd6V_cRw6iCI_-Unl3PQ',
    "server": "http://higlass.io/api/v1/",
    "height": 210,
    "options": {}
}

t1 = Track(**track_def)
t2 = Track(**{ **track_def, "tileset_uuid": "QvdMEvccQuOxKTEjrVL3wA" })
t3 = (t1 / t2).change_attributes(
    options={
        'colorRange': hg_cmap('coolwarm'),
        'valueScaleMin': 0.1,
        'valueScaleMax': 10,
    })
domain = [7e7,8e7]

v1 = View([t1], x=0, width=4, initialXDomain=domain)
v2 = View([t3], x=4, width=4, initialXDomain=domain)
v3 = View([t2], x=8, width=4, initialXDomain=domain)

display, server, viewconf = higlass.display([v1, v2, v3])
display

Saving the view

The currently visible HiGlass view can be downloaded to a file:

display.save_as_png('/tmp/my_view.png')

Not that this function can only be used within a Jupyter notebook and works asynchronously so the saved screenshot will not nessarily be complete immediately after the function finishes executing

Authorization

If loading tiles from a secured server, the auth_token parameter takes the string that will be used as the Authorization header on all tile requests sent out by HiGlass:

(d,s,v) = higlass.display(views, auth_token='JWT DEADBEEF')

Other Examples

The examples below demonstrate how to use the HiGlass Python API to view data locally in a Jupyter notebook or a browser-based HiGlass instance.

For a more complete overview, you can find the demos from the talk at github.com/higlass/scipy19.

Jupyter HiGlass Component

To instantiate a HiGlass component within a Jupyter notebook, we first need to specify which data should be loaded. This can be accomplished with the help of the higlass.client module:

from higlass.client import View, Track
import higlass


view1 = View([
    Track(track_type='top-axis', position='top'),
    Track(track_type='heatmap', position='center',
          tileset_uuid='CQMd6V_cRw6iCI_-Unl3PQ',
          server="http://higlass.io/api/v1/",
          height=250,
          options={ 'valueScaleMax': 0.5 }),
])

Remote bigWig Files

bigWig files can be loaded either from the local disk or from remote http servers. The example below demonstrates how to load a remote bigWig file from the UCSC genome browser’s archives. Note that this is a network-heavy operation that may take a long time to complete with a slow internet connection.

from higlass.client import View, Track
import higlass.tilesets

ts1 = higlass.tilesets.bigwig(
    'http://hgdownload.cse.ucsc.edu/goldenpath/hg19/encodeDCC/'
    'wgEncodeSydhTfbs/wgEncodeSydhTfbsGm12878InputStdSig.bigWig')

tr1 = Track('horizontal-bar', tileset=ts1)
view1 = View([tr1])
display, server, viewconf = higlass.display([view1])

display

Serving local data

To view local data, we need to define the tilesets and set up a temporary server.

Cooler Files

Creating the server:

from higlass.client import View, Track
from higlass.tilesets import cooler
import higlass

ts1 = cooler('../data/Dixon2012-J1-NcoI-R1-filtered.100kb.multires.cool')
tr1 = Track('heatmap', tileset=ts1)
view1 = View([tr1])
display, server, viewconf = higlass.display([view1])

display

BigWig Files

In this example, we’ll set up a server containing both a chromosome labels track and a bigwig track. Furthermore, the bigwig track will be ordered according to the chromosome info in the specified file.

from higlass.client import View, Track
from higlass.tilesets import bigwig, chromsizes
import higlass.tilesets

chromsizes_fp = '../data/chromSizes_hg19_reordered.tsv'
bigwig_fp = '../data/wgEncodeCaltechRnaSeqHuvecR1x75dTh1014IlnaPlusSignalRep2.bigWig'

with open(chromsizes_fp) as f:
    chromsizes = []
    for line in f.readlines():
        chrom, size = line.split('\t')
        chromsizes.append((chrom, int(size)))

cs = chromsizes(chromsizes)
ts = bigwig(bigwig_fp, chromsizes=chromsizes)

tr0 = Track('top-axis')
tr1 = Track('horizontal-bar', tileset=ts)
tr2 = Track('horizontal-chromosome-labels', position='top', tileset=cs)

view1 = View([tr0, tr1, tr2])
display, server, viewconf = higlass.display([view1])

display

The client view will be composed such that three tracks are visible. Two of them are served from the local server.

Serving custom data

To display data, we need to define a tileset. Tilesets define two functions: tileset_info:

> from higlass.tilesets import bigwig
> ts1 = bigwig('http://hgdownload.cse.ucsc.edu/goldenpath/hg19/encodeDCC/wgEncodeSydhTfbs/wgEncodeSydhTfbsGm12878InputStdSig.bigWig')
> ts1.tileset_info()
{
 'min_pos': [0],
 'max_pos': [4294967296],
 'max_width': 4294967296,
 'tile_size': 1024,
 'max_zoom': 22,
 'chromsizes': [['chr1', 249250621],
                ['chr2', 243199373],
                ...],
 'aggregation_modes': {'mean': {'name': 'Mean', 'value': 'mean'},
                       'min': {'name': 'Min', 'value': 'min'},
                       'max': {'name': 'Max', 'value': 'max'},
                       'std': {'name': 'Standard Deviation', 'value': 'std'}},
 'range_modes': {'minMax': {'name': 'Min-Max', 'value': 'minMax'},
                 'whisker': {'name': 'Whisker', 'value': 'whisker'}}
 }

and tiles:

> ts1.tiles(['x.0.0'])
[('x.0.0',
  {'min_value': 0.0,
   'max_value': 9.119079544037932,
   'dense': 'Rh25PwcCcz...',   # base64 string encoding the array of data
   'size': 1,
   'dtype': 'float32'})]

The tiles function will always take an array of tile ids of the form id.z.x[.y][.transform] where z is the zoom level, x is the tile’s x position, y is the tile’s y position (for 2D tilesets) and transform is some transform to be applied to the data (e.g. normalization types like ice).

Numpy Matrix

By way of example, let’s explore a numpy matrix by implementing the tileset_info and tiles functions described above. To start let’s make the matrix using the Eggholder function.

import numpy as np

dim = 2000
I, J = np.indices((dim, dim))
data = (
    -(J + 47) * np.sin(np.sqrt(np.abs(I / 2 + (J + 47))))
    - I * np.sin(np.sqrt(np.abs(I - (J + 47))))
)

Then we can define the data and tell the server how to render it.

from  clodius.tiles import npmatrix
from higlass.tilesets import Tileset

ts = Tileset(
    tileset_info=lambda: npmatrix.tileset_info(data),
    tiles=lambda tids: npmatrix.tiles_wrapper(data, tids)
)

display, server, viewconf = higlass.display([
    View([
        Track(track_type='top-axis', position='top'),
        Track(track_type='left-axis', position='left'),
        Track(track_type='heatmap',
              position='center',
              tileset=ts,
              height=250,
              options={ 'valueScaleMax': 0.5 }),

    ])
])
display

Displaying Many Points

To display, for example, a list of 1 million points in a HiGlass window inside of a Jupyter notebook. First we need to import the custom track type for displaying labelled points:

%%javascript

require(["https://unpkg.com/higlass-labelled-points-track@0.1.11/dist/higlass-labelled-points-track"],
    function(hglib) {

});

Then we have to set up a data server to output the data in “tiles”.

import numpy as np
import pandas as pd
from higlass.client import View, Track
from higlass.tilesets import dfpoints

length = int(1e6)
df = pd.DataFrame({
    'x': np.random.random((length,)),
    'y': np.random.random((length,)),
    'v': range(1, length+1),
})

ts = dfpoints(df, x_col='x', y_col='y')

display, server, viewconf = higlass.display([
    View([
        Track('left-axis'),
        Track('top-axis'),
        Track('labelled-points-track',
               tileset=ts,
               position='center',
               height=600,
               options={
                    'xField': 'x',
                    'yField': 'y',
                    'labelField': 'v'
        }),
    ])
])

display

This same technique can be used to display points in a GeoJSON file. First we have to extract the values from the GeoJSON file and create a dataframe:

import math

def lat2y(a):
  return 180.0/math.pi*math.log(math.tan(math.pi/4.0+a*(math.pi/180.0)/2.0))

x = [t['geometry']['coordinates'][0] for t in trees['features']]
y = [-lat2y(t['geometry']['coordinates'][1]) for t in trees['features']]
names = [t['properties']['SPECIES'] for t in trees['features']]

df = pd.DataFrame({ 'x': x, 'y': y, 'names': names })
df = df.sample(frac=1).reset_index(drop=True)

And then create the tileset and track, as before.

from higlass.client import View, Track
from higlass.tilesets import dfpoints

ts = dfpoints(df, x_col='x', y_col='y')

display, server, viewconf = higlass.display([
    View([
        Track('left-axis'),
        Track('top-axis'),
        Track('osm-tiles', position='center'),
        Track('labelled-points-track',
               tileset=ts,
               position='center',
               height=600,
               options={
                    'xField': 'x',
                    'yField': 'y',
                    'labelField': 'names'
        }),
    ])
])

display

Other constructs

The examples containing dense data above use the bundled_tiles_wrapper_2d function to translate lists of tile_ids to tile data. This consolidates tiles that are within rectangular blocks and fulfills them simultaneously. The return type is a list of (tile_id, formatted_tile_data) tuples.

In cases where we don’t have such a function handy, there’s the simpler tiles_wrapper_2d which expects the target to fullfill just single tile requests:

from clodius.tiles.format import format_dense_tile
from clodius.tiles.utils import tiles_wrapper_2d
from higlass.tilesets import Tileset

ts = Tileset(
    tileset_info=tileset_info,
    tiles=lambda tile_ids: tiles_wrapper_2d(tile_ids,
                    lambda z,x,y: format_dense_tile(tile_data(z, x, y)))
)

In this case, we expect tile_data to simply return a matrix of values.

Troubleshooting

Accessing the server log

A local server writes its log records to an in-memory StringIO buffer. The server’s name can be used to access its logger.

import logging

logger = logging.getLogger(server.name)
logger.info('Hi!')

# convert the stream into a string
print(server.log.getvalue())

# write the log to a file
with open('higlass-server.log', 'wt') as f:
    f.write(server.log.getvalue())