NOTE: The information below is expected to be modular, in that it can be reproduced locally or on the cloud. If running in MMC, it is recommended you first follow the steps in this page first in order to get your interactive setup working. Afterwards, all relevant information below needs to be done in the interactive tmate session

JupyterLab + SoS Suite setup

This document provides tips for setting up your JupyterLab + SoS Suite computing environment using pixi package manager.

Operating OS requirement

The instructions on this page are tested and known to work for Linux and MacOS. Although with some efforts it might work for Windows, using Windows your every day computational biology research is discouraged.

A note for Columbia Neurology HPC Users

To configure the network proxy, add the following commands to your ~/.bashrc and then run the source command. Begin by opening ~/.bashrc in a text editor and appending the commands:

export http_proxy=http://menloproxy.cumc.columbia.edu:8080
export https_proxy=http://menloproxy.cumc.columbia.edu:8080

and type source ~/.bashrc to load the changes.

Purge previous installations

This is an optional step only necessary for those who had installed various software previously and now would like to start from scratch. On MMC, it is recommended you ask an admin user to help clear the directory for you - in case you affect other people’s installations. Please follow the instructions here to start up your tmate session. You can find instructions to start your tmate session here.

On MMC:

cd /mnt/efs/$YOUR_USER_NAME
rm -rf ~/.mamba ~/.conda ~/.anaconda ~/.pixi ~/.jupyter

On other setups:

rm -rf ~/.mamba ~/.conda ~/.anaconda ~/.pixi ~/.jupyter

Basic software environment and manager setup

To install our customized environment based on pixi, please follow the instructions here.

At this point, your system is successfully installed with pixi. Depending on your needs, you can add extra software packages. The following section gives examples how to install other executables as well as R and Python packags.

A note to users from China

Depending on your network settings, sometimes you might experience difficulties following from this setup due to GitHub connectivity issues, this repository provides some walk-arounds to set it up using resources from Gitee instead.

Next steps for HPC users

You are now clear to start setting up for connecting to the HPC via JupyterLab.

Install other executables: use pixi

Once this is set, you can install other executables using pixi as the software manager, as long as they are released in one of the conda channels. Please remember this is done in your interactive tmate session. For example to install bcftools

pixi global install bcftools

and vim the text editor,

pixi global install vim

Note: don’t use pixi to install R or Python libraries (see below).

Install other R libraries: use pixi

It is important to realize that the R software and libraries here are included in this environment, using precompiled packages from conda installed via pixi. It is therefore highly recommended that R libraries be installed also from conda as long as they are available.

For example, to install R library pacman you can verify that it is available on conda-forge; then you can install it using:

pixi global install -c conda-forge --environment r-base r-pacman

to install it to the r_libs environment that we have previously configured.

For libraries not available on conda, you can use the regular approaches to install them, such as from cran, bioconductor and GitHub although it is strongly recommended to install from conda as long as it is possible.

Install other Python packages: use pixi

For example, to install Python library pacman you can verify that it is available on conda-forge; then you can install it using:

pixi global install -c conda-forge --environment python seaborn

to install it to the python_libs environment that we have previously configured.

For packages not available on conda, you can use the regular approaches to install them, such as from pypi via pip install; again try avoid doing that but rely on conda as much as possible.