Connect from your personal computer to another computer
Here we introduce in brief a few methods to connect and work with remote computers to perform data analysis. The remote computer can be a desktop computer or a HPC cluster system. I’ll call it “host” for the remote computer, and “guest” for the computer through which you connect to the host.
Mount shared drive to a Windows computer
- From Windows Explorer, right click on “This PC” and select “Map network drive”
- In the “Map Nework Drive” window enter the path to the share
\\prometheus.neuro.columbia.edu\hgrcgrid_statgenand select “Connect using different credentials” then “Finish”
- Enter your username as
nimaster\Unialong with your Neurology/work computer password.
(Above are notes from Thashi and Anu)
Through VS Code
Through Rstudio server
JupyterLab server on Columbia Neurology HPC
The goal is to setup a JupyterLab server so you can access an interactive analysis environment on the cluster from your local computer.
Install JupyterLab environment
Follow the instructions on this page to configure software on the cluster.
Start JupyterLab server on a compute node
Submit jupyter_columbia.sh as a job. We already have it on the cluster here(
Please copy it to your home folder, modify it as necessary then submit it:
- copy it to local account directory:
cp /mnt/vast/hpc/csg/pbs_template/jupyter_columbia.sh ~/
- modify it as necessary. modification to the template may include:
- Different requirement of memory and wall time
- Additional bash configurations before the command to launch jupyter-lab, eg if you follow the micromamba setup tutorial you will need to include additional configurations:
- submit it
qsub jupyter_columbia.sh, or to send it to a particular partition,
qsub -q csg2.q -l t_pri jupyter_columbia.sh, or to a particular node,
qlogin -q csg2.q -l t_pri -l h="node80" jupyter_columbia.sh
Create your ssh tunnel to access the server
Find information needed from the out file of the above submitted job (for example) :
‘831088’ is the job ID, which would be different for your case, replace it with your job ID.
Find the line looks like the following one in your output file, copy paste it into a new terminal window that is not connected to the cluster:
ssh -N -L 8476:node31:8476 <your UNI>@csglogin.neuro.columbia.edu
Access the server from web browser
Find information needed from the .out file of the above submitted job (for example) :
‘802790’ is the job ID, which would be different for your case, replace it with your job ID.
Copy the URL in this .out file and paste it into a browser to launch your jupyter lab.
Note: If your are using Windows WSL with its default terminal there is a chance that using
export PATH=$HOME/micromamba/.local/bin:$PATH gives “jupyter-lab command-not-found” error in the
.out file. Change it to
export PATH="/home/[your-uni]/micromamba/bin:$PATH" fixed the problem – you need to expand
JupyterLab server on RCC at UChicago
For University of Chicago RCC (last connected Spring 2020) we can run JupyterLab on compute nodes and connect to it directly via SSH tunnel, along the lines of this post. It does not involve running anything on the login node.
Connect to a stand alone desktop or cloud VM through command shell
For other standalone Linux and Mac systems in the same network (we no longer have this set up in the lab any more — you can skip it).
Get IP address
First, obtain the IP address or domain name of the host computer. You may either be provided with the information, or you can run in command terminal (for a Linux host):
to see the IP address. For example:
$ sudo ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.105 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::ae1f:6bff:fe11:ab10 prefixlen 64 scopeid 0x20<link> ether ac:1f:6b:11:ab:10 txqueuelen 1000 (Ethernet) RX packets 123292 bytes 105881116 (100.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 130328 bytes 41654696 (39.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xc7200000-c727ffff eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether ac:1f:6b:11:ab:11 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xc7100000-c717ffff
The information you should be looking at is the
inet property under
eth0, which in our case is
(this is the IP address in my local network domain).
Connect via SSH
On the guest computer, type
username is the account name you have on the host machine. Please replace
192.168.0.105 with your IP address.
If it fails, it might be possible that your host machine does not support
SSH access. In that case you need to install SSH server on your host. For a Debian / Ubuntu host:
sudo apt-get install openssh-server
If you want to be able to use graphical interface for apps on your host, for example use
firefox to browse internet from host, you need to
-X option, eg,
ssh -X email@example.com
Create a user on the host machine
If you do not have a user account on the host, ask whoever administrates that machine to create one for you. He / she can use the command below:
username=hxl group=docker # another option is `www-data` group, for a web server cd /home sudo useradd -g $group $username -d /home/$username -m -s /bin/bash sudo passwd $username sudo cp .bash_profile .bash_prompt /home/$username sudo echo 'export LC_ALL="en_US.UTF-8"' >> /home/$username/.bashrc sudo chown $username.$group -R /home/$username
You’ll be prompt to enter password. This will add user
Notice the line
cp .bash_profile .bash_prompt /home/$username – I put a couple of template configuration files under
/home folder to copy to a new account. This will make the terminal prompt look better.
Connect to a cloud VM through command shell for Windows 10
Windows 10 users can install the
WSL program to have a Linux subsystem on their Windows computer. Then can connect following the same steps as above for Linux users.
Connect to a cloud VM through JupyterLab
This is a more generic tutorial. Please focus on the first section of this document if you are only interested in connecting to Columbia Neurology HPC via JupyterLab.
Install JupyterLab on host
If you are a JupyterLab user you probably already have JupyterLab running on the host, or at least knows how to install it. If not, this page contains the commands to set it up.
Start JupyterLab server on host
First you need to connect to the host via SSH. Please open a new terminal to connect. Details see section above.
Once you are connected to host, type this line:
jupyter-lab --no-browser &
You should see lots of output. Of relevance is some lines like this:
[C 12:40:51.496 LabApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=1a90c5465f3295c8354a6e2365b961e88affb880559f51cb
Once you see it you know the JupyterLab server is up and running (on the background on the remote host). There are two important piece of information you need to keep:
- The port number the JupyterLab server is to be accessed:
- The token (password):
The port number and token can be different every time you connect.
At this point you can hit
enter key to bring yourself back to the terminal, then type
exit to quit the host. It is very important to quit properly!
Connect to host JupyterLab from guest
On your guest machine, type:
ssh -NL 1828:localhost:8888 firstname.lastname@example.org
8888is the port number you see in the previous step on the host. Replace it with what you actually see.
usernameis your username on the host, replace it with your username
192.168.0.105is my host IP address, see previous section on SSH connections. Replace it with the actual IP of your host computer.
1828is a port number you assign on your guest machine to map to host JupyterLab server port. It can be a random number like this, or like
6666as long is they are not being used by other apps on your computer. Just type in any 4-digits number. The command will complain and fail if the port you choose is not available.
enter, you might be asked to enter password. After that, nothing will show up. Do not worry it is a good sign! Now in your web browser type
http://localhost:1828 you should be directed to the page where you get asked the token and you input that using the token generated above.
For Google Cloud users
Here is a Youtube video tutorial on how to connect to Google Cloud VM through JupyterLab.
Use command terminal from within JupyterLab
The “Launcher” page of JupyterLab has a few types of items to launch:
We typically launch a
Notebook, but the other two are also very useful. For example to get yourself a conventional command terminal you just click on
Terminal then you’ll be brought to the command terminal. You can also try to create a Bash
Console. It is like something in between terminal and notebook. Sometimes you might find it more user-friendly.