JupyterLab + SoS Suite setup

This document provides tips for setting up your JupyterLab + SoS Suite computing environment using pixi and micromamba package manager.

Operating OS requirement

The instructions on this page are tested and known to work for Linux and MacOS. Although with some efforts it might work for Windows, using Windows your every day computational biology research is discouraged.

Note for Neurology HPC Users: To configure the network proxy, add the following commands to your ~/.bashrc and then run the source command. Begin by opening ~/.bashrc in a text editor and appending the commands:

export http_proxy=http://menloproxy.cumc.columbia.edu:8080
export https_proxy=http://menloproxy.cumc.columbia.edu:8080

and type source ~/.bashrc to load the changes.

Purge previous installations of micromamba or miniconda

This is an optional step only necessary for those who had installed various software previously and now would like to start from scratch.

First, back-up the previous micromamba installation by:

mv ~/micromamba ~/micromamba_backup
mv ~/.conda ~/conda_backup
rm -rf ~/.mamba ~/.conda ~/.anaconda ~/.pixi ~/.jupyter

For HPC users make a back up of ~/.bashrc by:

mv ~/.bashrc ~/.bashrc_backup

and install a new copy of it:

cp /etc/skel/.bashrc ~/.bashrc

For MacOS users this file might be ~/.zshrc or ~/.bash_profile depending on your shell setup.

Finally, open up your ~/.bashrc_backup, review and move the contents that you deem relevant to the new environment that you are about to setup. For example, the http_proxy and https_proxy discussed in the previous section should be retained for Neurology HPC users. However if you would like micromamba to setup R and not using R installed on the HPC, please do not include module load R in the new bashrc that you are configuring now.

At this point, please log out then log back in to refresh the computing environment.

Initial setup

To install our customized environment based on pixi and micromamba,

curl -fsSL https://raw.githubusercontent.com/gaow/misc/master/bash/pixi/pixi-setup.sh | bash

At this point, you can test your installation by connecting to the HPC via JupyterLab. If everything works well so far, you can optionally delete your back-up folders for the previous setup:

rm -rf ~/micromamba_backup
rm -rf ~/conda_backup

Install other executables: use pixi

Once this is set, you can install other executables using pixi as the software manager, as long as they are released in one of the conda channels. For example to install bcftools

pixi global install bcftools

and vim the text editor,

pixi global install vim

Note: don’t use pixi to install R or Python libraries (see below).

Install other R libraries: use micromamba

It is important to realize that the R software and libraries here are included in this environment, using precompiled packages from conda installed via micromamba. It is therefore highly recommended that R libraries be installed also from conda as long as they are available.

For example, to install R library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n r_libs r-pacman -c conda-forge -y

to install it to the r_libs environment that we have previously configured.

For libraries not available on conda, you can use the regular approaches to install them, such as from cran, bioconductor and GitHub although it is strongly recommended to install from conda as long as it is possible.

Install other Python packages: use micromamba

For example, to install Python library pacman you can verify that it is available on conda-forge; then you can install it using:

micromamba install -n python_libs seaborn -c conda-forge -y

to install it to the python_libs environment that we have previously configured.

For packages not available on conda, you can use the regular approaches to install them, such as from pypi via pip install; again try avoid doing that but rely on conda as much as possible.