JupyterLab + SoS Suite setup

This document provides tips for setting up your JupyterLab + SoS Suite computing environment using pixi and micromamba package manager.

Operating OS requirement

The instructions on this page are tested and known to work for Linux and MacOS. Although with some efforts it might work for Windows, using Windows your every day computational biology research is discouraged.

For those using a Debian-based Linux desktop (e.g., Debian or Ubuntu), you can find setup recommendations here.
MacOS users with version 11.X or above can find setup recommendations here.
If you only have access to Windows, you might consider establishing a Linux OS within your Windows system using the Windows Subsystem for Linux (WSL). For assistance, here’s a video tutorial by one of our lab members on how to install WSL on a Windows machine.

A note for Columbia Neurology HPC Users

To configure the network proxy, add the following commands to your ~/.bashrc and then run the source command. Begin by opening ~/.bashrc in a text editor and appending the commands:

export http_proxy=http://menloproxy.cumc.columbia.edu:8080
export https_proxy=http://menloproxy.cumc.columbia.edu:8080

and type source ~/.bashrc to load the changes.

Purge previous installations of `micromamba` or `miniconda`

This is an optional step only necessary for those who had installed various software previously and now would like to start from scratch.

rm -rf ~/.mamba ~/.conda ~/.anaconda ~/.pixi ~/.jupyter

For Linux users make a back up of ~/.bashrc by:

mv ~/.bashrc ~/.bashrc_backup

and install a new copy of it:

cp /etc/skel/.bashrc ~/.bashrc

For MacOS users this file might be ~/.zshrc or ~/.bash_profile depending on your shell setup.

Finally, open up your ~/.bashrc_backup, review and move the contents that you deem relevant to the new environment that you are about to setup. For example, the http_proxy and https_proxy discussed in the previous section should be retained for Neurology HPC users. However if you would like micromamba to setup R and not using R installed on the HPC, please do not include module load R in the new bashrc that you are configuring now.

At this point, please log out then log back in to refresh the computing environment.

Basic software environment and manager setup

To install our customized environment based on pixi and micromamba,

curl -fsSL https://raw.githubusercontent.com/gaow/misc/master/bash/pixi/pixi-mamba.sh | bash

At this point, your system is successfully installed with pixi and micromamba. Depending on your needs, you can add extra software packages. The following section gives examples how to install other executables as well as R and Python packags.

A note to users from China

Depending on your network settings, sometimes you might experience difficulties following from this setup due to GitHub connectivity issues, this repository provides some walk-arounds to set it up using resources from Gitee instead.

Next steps for HPC users

You are now clear to start setting up for connecting to the HPC via JupyterLab.

Install other executables: use `pixi`

Once this is set, you can install other executables using pixi as the software manager, as long as they are released in one of the conda channels. For example to install bcftools

pixi global install bcftools

and vim the text editor,

pixi global install vim

Note: don’t use pixi to install R or Python libraries (see below).

Install other R libraries: use `micromamba`

It is important to realize that the R software and libraries here are included in this environment, using precompiled packages from conda installed via micromamba. It is therefore highly recommended that R libraries be installed also from conda as long as they are available.

For example, to install R library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n r_libs r-pacman -c conda-forge -y

to install it to the r_libs environment that we have previously configured.

For libraries not available on conda, you can use the regular approaches to install them, such as from cran, bioconductor and GitHub although it is strongly recommended to install from conda as long as it is possible.

Install other Python packages: use `micromamba`

For example, to install Python library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n python_libs seaborn -c conda-forge -y

to install it to the python_libs environment that we have previously configured.

For packages not available on conda, you can use the regular approaches to install them, such as from pypi via pip install; again try avoid doing that but rely on conda as much as possible.