Exercise: Jupyter Notebook on Midway via SSH

Exercise: Jupyter Notebook on Midway via SSH#

In this exercise, you will:

  1. Connect to Midway3 via SSH

  2. Request an interactive compute node with SLURM

  3. Launch a Jupyter notebook server on the compute node

  4. Set up SSH port forwarding so you can access the notebook from your laptop’s browser

By the end, you’ll have a Jupyter notebook running on a powerful remote machine, accessible from your laptop as if it were running locally.

Prerequisites#

  • A UChicago CNetID with RCC access

  • A terminal application (Terminal on macOS, Windows Terminal or Git Bash on Windows)

  • Duo Mobile configured for multi-factor authentication

Step 1: SSH into Midway3#

Open a terminal on your laptop and connect:

ssh <your-cnetid>@midway3.rcc.uchicago.edu

Enter your password when prompted, then complete Duo MFA. You should see a welcome message and a prompt like:

[<cnetid>@midway3-login1 ~]$

You are now on a login node.

Reminder

Do not run computations on the login node. It is shared by all users and intended only for lightweight tasks like file editing, job submission, and data transfer.

Step 2: Request an Interactive Compute Node#

Use sinteractive to request a dedicated compute node:

sinteractive --account=finm32900 --partition=caslake \
    --time=02:00:00 --cpus-per-task=4 --mem=16G

This requests:

  • Account: finm32900 (our course allocation)

  • Partition: caslake (the default Midway3 partition)

  • Time: 2 hours

  • CPUs: 4 cores

  • Memory: 16 GB

You may wait briefly in the queue. Once a node is allocated, your prompt changes to show the compute node name:

[<cnetid>@midway3-0042 ~]$

You are now on a compute node with dedicated resources.

Step 3: Set Up the Python Environment#

Load the Python module and activate the course virtual environment:

module load python/3.11.9
source /project/finm32900/tdp-venv-p3119/bin/activate

Verify that Jupyter is available:

which jupyter-notebook

Tip

If you want to use your own virtual environment instead, create one in /project/finm32900/<cnetid>/ or /scratch/midway3/<cnetid>/ (not in /home/, which has a small quota).

Step 4: Get the Host IP and Start Jupyter#

The compute node has an internal IP address that we’ll need for port forwarding. Get it and start Jupyter:

# Get the compute node's IP address
HOST_IP=$(/sbin/ip route get 8.8.8.8 | awk '{print $7;exit}')

# Pick a random port (to avoid collisions with other users)
PORT_NUM=$(shuf -i15001-30000 -n1)

# Print the connection info (you'll need this in Step 5)
echo "=========================================="
echo "  Host IP:  $HOST_IP"
echo "  Port:     $PORT_NUM"
echo "=========================================="

# Start Jupyter
jupyter-notebook --no-browser --ip=$HOST_IP --port=$PORT_NUM

Jupyter will print output like:

[I 14:23:07.891 NotebookApp] Serving notebooks from local directory: /home/<cnetid>
[I 14:23:07.891 NotebookApp] Jupyter Notebook is running at:
[I 14:23:07.891 NotebookApp]   http://10.50.221.42:18273/?token=abc123def456...

Important: Note the connection details

Write down or copy:

  • The Host IP (e.g., 10.50.221.42)

  • The Port (e.g., 18273)

  • The token from the URL

You will need all three in the next step. Leave this terminal open—Jupyter is running in the foreground.

Step 5: Set Up SSH Port Forwarding#

Open a new terminal window on your laptop (do not close the Jupyter terminal). Run:

ssh -N -f -L <PORT_NUM>:<HOST_IP>:<PORT_NUM> \
    <your-cnetid>@midway3.rcc.uchicago.edu

Replace <PORT_NUM> and <HOST_IP> with the values from Step 4. For example:

ssh -N -f -L 18273:10.50.221.42:18273 jdoe@midway3.rcc.uchicago.edu

Complete Duo MFA when prompted. The command runs in the background (no visible output means success).

What this does:

Your laptop (localhost:18273)
     │
     │  SSH tunnel (encrypted)
     ▌
Midway3 login node
     │
     │  Internal network
     ▌
Compute node (10.50.221.42:18273)  ← Jupyter is running here

Traffic to localhost:18273 on your laptop is forwarded through the SSH tunnel to the Jupyter server on the compute node.

Step 6: Open Jupyter in Your Browser#

On your laptop, open a web browser and go to:

http://127.0.0.1:<PORT_NUM>/?token=<your-token>

For example:

http://127.0.0.1:18273/?token=abc123def456...

You should see the Jupyter notebook interface. You are now running code on a Midway3 compute node with 4 CPU cores and 16 GB of RAM, but interacting with it through your laptop’s browser.

Step 7: Test It#

Create a new Python 3 notebook and run:

import os
import platform

print(f"Hostname: {platform.node()}")
print(f"CPUs available: {os.cpu_count()}")
print(f"User: {os.environ.get('USER', 'unknown')}")

You should see the compute node’s hostname (e.g., midway3-0042), confirming that your code is running on the cluster, not your laptop.

Try something more substantial:

import numpy as np

# Generate a large random matrix (uses cluster RAM, not your laptop's)
X = np.random.randn(10_000, 10_000)
print(f"Matrix shape: {X.shape}")
print(f"Memory used: {X.nbytes / 1e9:.1f} GB")

# Matrix multiplication (uses cluster CPUs)
result = X @ X.T
print(f"Computation complete. Result shape: {result.shape}")

Step 8: Clean Up#

When you’re done:

  1. Save your notebooks (File → Save).

  2. Stop Jupyter: Go to the terminal where Jupyter is running and press Ctrl+C, then confirm with y.

  3. Exit the compute node: Type exit to release the node back to the cluster.

  4. Kill the SSH tunnel (optional): Find and kill the background SSH process on your laptop:

    # Find the tunnel process
ps aux | grep "ssh -N -f -L"

# Kill it by PID
kill <pid>

Always clean up

When you’re done working, release the compute node by exiting the sinteractive session. Idle nodes still consume your account’s allocation (SUs). Don’t leave sessions running overnight.

Troubleshooting#

“Port already in use”#

If Jupyter says the port is already in use, pick a different random port:

PORT_NUM=$(shuf -i15001-30000 -n1)
echo "New port: $PORT_NUM"
jupyter-notebook --no-browser --ip=$HOST_IP --port=$PORT_NUM

SSH tunnel fails or times out#

  • Make sure you completed Duo MFA for the tunnel connection.

  • Check that the HOST_IP and PORT_NUM match exactly between Jupyter and the tunnel command.

  • Try killing any existing tunnel processes and creating a new one.

“Connection refused” in browser#

  • Verify Jupyter is still running in the first terminal.

  • Verify the SSH tunnel is active: ps aux | grep "ssh -N -f -L".

  • Make sure the port numbers match in all three places: Jupyter startup, SSH tunnel command, and browser URL.

Jupyter exits immediately#

  • Check that you loaded the Python module (module load python/3.11.9) and activated the virtual environment before starting Jupyter.

  • Run which jupyter-notebook to confirm it’s available.

Queue wait is too long#

  • Try the debug QoS for a quick 15-minute session:

    sinteractive --account=finm32900 --qos=debug --time=00:15:00

  • Check the queue: squeue --partition=caslake to see how busy the partition is.

Summary#

You’ve now completed the full workflow for running Jupyter on an HPC cluster:

  1. SSH into the cluster login node

  2. SLURM (sinteractive) to get a compute node

  3. Jupyter started on the compute node

  4. SSH port forwarding to tunnel the connection to your laptop

  5. Browser to interact with the notebook

This same pattern works for any web-based tool running on a remote machine, not just Jupyter. Any time you need to access a service running on a port of a remote machine, SSH port forwarding is the solution.

Contents