JupyterLab + SoS Suite setup

This document provides tips for setting up your JupyterLab + SoS Suite computing environment using pixi and micromamba package manager.

Operating OS requirement

The instructions on this page are tested and known to work for Linux and MacOS. Although with some efforts it might work for Windows, using Windows your every day computational biology research is discouraged.

A note for Columbia Neurology HPC Users

To configure the network proxy, add the following commands to your ~/.bashrc and then run the source command. Begin by opening ~/.bashrc in a text editor and appending the commands:

export http_proxy=http://menloproxy.cumc.columbia.edu:8080
export https_proxy=http://menloproxy.cumc.columbia.edu:8080

and type source ~/.bashrc to load the changes.

Purge previous installations of micromamba or miniconda

This is an optional step only necessary for those who had installed various software previously and now would like to start from scratch.

First, back-up the previous micromamba installation by:

mv ~/micromamba ~/micromamba_backup
mv ~/.conda ~/conda_backup
rm -rf ~/.mamba ~/.conda ~/.anaconda ~/.pixi ~/.jupyter

For Linux users make a back up of ~/.bashrc by:

mv ~/.bashrc ~/.bashrc_backup

and install a new copy of it:

cp /etc/skel/.bashrc ~/.bashrc

For MacOS users this file might be ~/.zshrc or ~/.bash_profile depending on your shell setup.

Finally, open up your ~/.bashrc_backup, review and move the contents that you deem relevant to the new environment that you are about to setup. For example, the http_proxy and https_proxy discussed in the previous section should be retained for Neurology HPC users. However if you would like micromamba to setup R and not using R installed on the HPC, please do not include module load R in the new bashrc that you are configuring now.

At this point, please log out then log back in to refresh the computing environment.

Basic software environment and manager setup

To install our customized environment based on pixi and micromamba,

curl -fsSL https://raw.githubusercontent.com/gaow/misc/master/bash/pixi/pixi-setup.sh | bash

Now you can start using the system. If you eventually decide that everything works well, you can delete your back-up folders for the previous setup:

rm -rf ~/micromamba_backup
rm -rf ~/conda_backup

At this point, your system is successfully installed with pixi and micromamba. Depending on your needs, you can add extra software packages. The following section gives examples how to install other executables as well as R and Python packags.

A note to users from China

Depending on your network settings, sometimes you might experience difficulties following from this setup due to GitHub connectivity issues, this repository provides some walk-arounds to set it up using resources from Gitee instead.

Next steps for HPC users

You are now clear to start setting up for connecting to the HPC via JupyterLab.

Install other executables: use pixi

Once this is set, you can install other executables using pixi as the software manager, as long as they are released in one of the conda channels. For example to install bcftools

pixi global install bcftools

and vim the text editor,

pixi global install vim

Note: don’t use pixi to install R or Python libraries (see below).

Install other R libraries: use micromamba

It is important to realize that the R software and libraries here are included in this environment, using precompiled packages from conda installed via micromamba. It is therefore highly recommended that R libraries be installed also from conda as long as they are available.

For example, to install R library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n r_libs r-pacman -c conda-forge -y

to install it to the r_libs environment that we have previously configured.

For libraries not available on conda, you can use the regular approaches to install them, such as from cran, bioconductor and GitHub although it is strongly recommended to install from conda as long as it is possible.

Install other Python packages: use micromamba

For example, to install Python library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n python_libs seaborn -c conda-forge -y

to install it to the python_libs environment that we have previously configured.

For packages not available on conda, you can use the regular approaches to install them, such as from pypi via pip install; again try avoid doing that but rely on conda as much as possible.