JupyterLab on HPC

Prerequisites

Either use module load or follow the instructions on the Software Setup page to configure your environment.

Start JupyterLab on a Compute Node

Download the JupyterLab script to your home directory:

curl -O https://wanggroup.org/scripts/jupyter_cumc_cloud.sh

Modify the script as needed:

• Adjust memory and wall time requirements: Default to 15GB memory running 18 hours which typically works for moderate scale of computing.
• Add bash configurations before the jupyter-lab command (e.g., custom paths from your ~/.bashrc or simply add source ~/.bashrc).

Submit the job:

sbatch jupyter_cumc_cloud.sh -i <hpc_ip>

To request specific resources:

sbatch --constraint="cpu4mem32" --time=24:00:00 jupyter_cumc_cloud.sh -i <hpc_ip>

Access the Node via SSH Tunnel

After the job starts, check the job output file for connection info:

cat jupyter_<JOBID>.out

Replace <JOBID> with your actual job ID. You’ll see instruction like:

========================================
Jupyter Lab - Job 12345678
========================================
On your LOCAL machine, run:
  ssh -N -L 8234:compute-node:8234 username@hpc_ip

Then find the Jupyter URL with token in:
  cat jupyter_12345678.log | grep "lab?token"

Copy the full URL (including token) to your browser.
To stop: scancel 12345678
========================================

Windows MobaXterm info:
  Forwarded port: 8234
  Remote server:  compute-node
  Remote port:    8234
  SSH server:     hpc_ip
  SSH login:      username
  SSH port:       22

========================================

Follow these instructions on your local machine to connect to JupterLab server.

Cost saving: Please remember to cancel your job if you are done with it.

Troubleshooting

• If browser complains about secure connection, please make sure http:// is used instead of https://
• If using Windows WSL with the default terminal, jupyter-lab command not found error may occur because $HOME is not expanded correctly.

Change:

export PATH=$HOME/.pixi/bin:$PATH

To:

export PATH="/home/<username>/.pixi/bin:$PATH"

Replace <username> with your actual username.