xQTL protocol on CAVATICA

The lab maintains a ready-to-run CAVATICA project, xqtl-protocol, which already contains the full xQTL protocol repository, all 104 pipeline notebooks, a packed conda/pixi environment and toy input data. Nothing needs to be built, installed or cloned, and there is no need to contact an admin. Each new Data Studio session needs exactly one setup command, which takes about 50 seconds.

1. Start a Data Studio session

1. Open the project on CAVATICA.
2. Click the Data Studio tab.
3. Click Start on an existing analysis, or Create new analysis, then JupyterLab, then attach this project, then Start.
4. Wait for the status to become RUNNING, then click Open.
5. In the JupyterLab launcher, under Other, click Terminal.

2. Set up the session

Run this once in every new session:

source /sbgenomics/project-files/xqtl-setup/setup_session.sh

Use source, not bash. The script exports variables into your shell and bash would throw them away. It takes about 50 seconds and reports unpacking of the pixi environment, installation of the python helper packages, the working directory /sbgenomics/workspace/xqtl-work, and that 104 pipeline notebooks are available. That is the entire environment setup.

3. See what you can run

ls $XQTL_PIPELINE/*.ipynb

To see the workflows and options of a single notebook:

$XQTL_SOS run $XQTL_PIPELINE/gene_annotation.ipynb -h

A pre-generated index of every notebook and its workflows is kept in $XQTL_SETUP/PIPELINES.md, which you can open in JupyterLab or read in the terminal with less.

4. Run the built-in demo

bash $XQTL_SETUP/run_mwe_chain.sh /sbgenomics/workspace/mwe_out

This takes about a minute and chains three real workflows on the toy proteomics data: annotate_coord from gene_annotation.ipynb (python), phenotype_by_chrom from phenotype_formatting.ipynb (python), and gEBMF from phenotype_imputation.ipynb (R). It prints ALL_THREE_STEPS_OK on success. Known-good reference results for comparison are in the project under xqtl-mwe/reference_output/.

5. Run any other pipeline

One generic wrapper runs any of the 104 notebooks:

bash $XQTL_SETUP/run_pipeline.sh [--py|--r] <notebook> <workflow> <outdir> [sos options]

--py is the default and is for python workflows, --r is for R workflows. Picking the wrong one gives a missing-module or missing-library error, so simply rerun with the other flag. The notebook may be a bare name, resolved inside $XQTL_PIPELINE, or a full path. The output directory is created for you. Example input data for many pipelines is already in $XQTL_INPUT.

An R example:

bash $XQTL_SETUP/run_pipeline.sh --r phenotype_imputation.ipynb gEBMF \
  /sbgenomics/workspace/imp_out \
  --phenoFile $XQTL_INPUT/proteomics/protocol_example.protein.missing.bed.gz \
  --num_factor 30

A python example:

bash $XQTL_SETUP/run_pipeline.sh gene_annotation.ipynb annotate_coord \
  /sbgenomics/workspace/anno_out \
  --phenoFile $XQTL_INPUT/rnaseq/protocol_example.rnaseq.bed.gz \
  --coordinate-annotation $XQTL_INPUT/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf \
  --phenotype-id-column gene_id

6. Keep your results

There are two writable locations and they behave very differently.

So either write straight into output-files:

bash $XQTL_SETUP/run_mwe_chain.sh /sbgenomics/output-files/my_results

or copy results there before you stop the session:

cp -r /sbgenomics/workspace/mwe_out /sbgenomics/output-files/

7. Stop the analysis

Click Stop at the top right of the JupyterLab window, click Confirm, and wait until the status goes from SAVING to SAVED. Stopping uploads everything in /sbgenomics/output-files into the project Files tab and stops the instance from billing, so always stop when you are done.

Downloading the demo data without a session

You do not need to start a Data Studio session just to get the example data. Open the project Files tab, tick the checkbox next to a file or folder and click Download. The demo input is under input/proteomics/ and the demo output under xqtl-mwe/reference_output/.

Optional: polyfun

Some fine-mapping pipelines need polyfun, which ships as a tarball to keep the project tidy. Unpack it into your session when you need it:

bash $XQTL_SETUP/unpack_polyfun.sh

It finishes in a few seconds and prints the location, for example /tmp/xqtl/polyfun. Pass that path to whichever pipeline needs it.

What is in the project

Variables set by setup_session.sh

Troubleshooting

Status and known limits

Last verified 2026-07-28 on a cold session: setup_session.sh completed in 49.7 seconds, all 104 pipeline notebooks loaded successfully (104 OK, 0 failed), and the three-step demo ran end to end and matched the reference output.