Use Singularity on HPC

There are two ways to build a singularity image: to build from source, or to convert from an existing docker image. But first of all, for Columbia Neurology HPC you may need to make following adjustment to your environment: put these lines in ~/.bashrc:

export SINGULARITY_TMPDIR="$HOME/tmp"
export SINGULARITY_LOCALCACHEDIR="$HOME/tmp"

and create a tmp folder in your home directory:

mkdir -p ~/tmp

This is because when singularity builds it needs enough disk space for temporary files. Default location is /tmp on the cluster node which may not have enough space for it when you have a large image to build.

Build from source

If you have a singularity configuration file *.def file, this section shows how to build a singularity image from it.

Convert docker configuration to singularity configuration file

If you have a docker configuration dockerfile instead of singularity def file, you can use the following commands to convert it to singularity configuration:

spython recipe {docker_config}.dockerfile | sed 's/Stage: spython-base//g' &> {singularity_config}.def

The spython program can be installed via pip install spython.

Build using --fakeroot:

singularity build --fakeroot {output_image}.sif {config_file}.def

If following errors occur:

FATAL:   could not use fakeroot: no mapping entry found in /etc/subuid for {uni}

or

ERROR  : Failed to create container process: Operation not permitted

you should take a look into the /etc/subuid file, see if you can find your UNI in it:

cat /etc/subuid

If not, please reach out to Jose (jmq2108), mention that you have this issue with Singularity, and request adding your UNI to the file /etc/subuid on the cluster to solve the problem.

Build using --remote:

Alternatively, you can build the image on the cloud. After following the prompt from singularity remote login to set up the account, the same sif image can be built by:

singularity build --remote {output_image}.sif {config_file}.def

Build from docker image

This is not recommended, unless you do not have the configuration file and have to rely on an existing docker image to create your singularity image.

Docker images on dockerhub.com can be pulled using command singularity pull docker://<hub-user>/<repo-name>. For example

singularity pull docker://gaow/twas

If singularity fail to convert from docker image due to memory constrains on your local machine, it is also possible to build the image remotely. First run singularity remote login and follow the prompt to set up your account, then

singularity build --remote twas.sif docker://gaow/twas:latest

The image will be built on the cloud then automatically downloaded to your local machine.

Using a singularity software environment from a Jupyter Notebook

This section, based on this tutorial, will cover basic strategies for using your singularity environment in a Jupyterlab server. Before proceed, please confirm that your singularity image has ipython kernel installed. This is usually true if your sif is based on a popular base image we use in the lab, gaow/base-notebook.

Custom kernels

IPython notebooks interface with the system via an abstraction called _Kernels_. It supports a wide variety of programming languages, and they can be customized by editing the kernelspec JSON file that defines them. Here is the default Python 3 kernelspec for reference:

"argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

The argv key in this JSON object is the list that Jupyter uses to construct the kernel command when a notebook is started.

A working example configuration for our HPC is:

{
"language": "python",
 "argv": [
  "/mnt/mfs/cluster/bin/Singularity-ce-3.9.4/bin/singularity",
   "exec",
   "-B",
   "/mnt/:/mnt/",
   "/home/hs3163/GIT/xqtl-pipeline/container/singularity/stephenslab.sif",
   "/opt/conda/bin/python",
   "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Singularity StephensLab tools",
 "metadata": {
  "debugger": true
 }
}

Another example for using R Kernel would be as follows, noted that instead of -m ipykernel -f, --slave -e IRkernel::main() --args are used.

{
"language": "R",
 "argv": [
  "/mnt/mfs/cluster/bin/Singularity-ce-3.9.4/bin/singularity",
   "exec",
   "-B",
   "/:/",
   "/home/hs3163/GIT/xqtl-pipeline/container/singularity/stephenslab.sif",
   "/opt/conda/envs/stephenslab/bin/R",
   "--slave -e",
  "IRkernel::main()",
  "--args",
  "{connection_file}"
 ],
 "display_name": "Singularity StephensLab R tools",
 "metadata": {
  "debugger": true
 }
}

In this example, only thing that needs to be changed is the /home/hs3163/GIT/xqtl-pipeline/container/singularity/stephenslab.sif to your specific container, and the display_name. I have set the binding point of singularity as “/mnt/:/mnt/” to best accommodate people’s needs.

Generating a new kernel

We’ll start by generating a new kernelspec in a temporary location under your home directory:

ipython kernel install --prefix ~/tmp

the output should look like:

[InstallIPythonKernelSpecApp] WARNING | Installing to /tmp/share/jupyter/kernels, which is not in ['/home/vagrant/.local/share/jupyter/kernels', '/home/vagrant/venv-jupyter/share/jupyter/kernels', '/usr/local/share/jupyter/kernels', '/usr/share/jupyter/kernels', '/home/vagrant/.ipython/kernels']. The kernelspec may not be found.
Installed kernelspec python3 in /tmp/share/jupyter/kernels/python3

Editing the kernel

Now replacing your kernelspec’s kernal.json file with the one prepared above. Make sure to rename the kernelspec directory to avoid conflicts with existing kernels, for example:

mv ~/tmp/share/jupyter/kernels/python3 ~/tmp/share/jupyter/kernels/stephenslab

Install as the jupyter kernel

Finish by installing your new kernel to a location where your notebook will look when it starts. The --user flag specifies that you wish to install the kernel only for your user, and prevents the install from attempting to use sys.prefix.

jupyter kernelspec install --user ~/tmp/share/jupyter/kernels/stephenslab

You should now be able to use it in JupyterLab as a kernel. If this new kernel is not available from your Jupyterlab, try disconnect and reconnect, or resubmitted another jupyterlab job so that the kernel are properly updated.

Caution: Using singularity software in this manner may create bugs that will not be encountered if using the singularity directly, as examplified here.Be sure to verify the bug within the container directly, i.e. not in the jupyter kernel .

Contact

Hao Sun