Week 9: Medium-Sized Data and Polars

• Strategies for Medium-Sized Data
• Polars Exercises: Code Snippets
• Remote Machines and High-Performance Computing
• Exercise: Jupyter Notebook on Midway via SSH
• Data Sources Overview
• Cleaning TRACE Corporate Bond Data: A Walkthrough

Agenda

• Final project proposal presentations: Ashish + Omar, Piyush, Ahmad + Jeffrey

• Medium-sized data strategies and Polars

• Introduction to TRACE: data sources overview and the Clean TRACE cleaning walkthrough

• Launch the reworked HW 4: deploy a live, self-updating FedWatch monitor (due Sunday, August 23)

• Final project logistics: rubric walkthrough, oral defense expectations, and signing up for a final presentation time (all groups must present by August 21)

Learning Outcomes

• Understand strategies for working with medium-sized datasets (1GB-100GB)

• Compare Pandas and Polars for data processing at scale

• Understand lazy evaluation, predicate pushdown, streaming, and Hive partitioning

• Introduction to TRACE corporate bond data

• Understand why data pipelines must decouple internet-dependent pulls from processing

Make-Up Material

Due to RCC access issues, the discussion of remote machines and HPC—including Remote Machines and HPC and the Exercise: Jupyter on Midway—is postponed. We will cover this material in a make-up session at a later date. Relatedly, Homework 5, which runs the Clean TRACE pipeline on RCC, is now optional.

When we cover this material, you will:

• Connect to remote machines via SSH and transfer files with rsync

• Understand HPC cluster architecture (login nodes, compute nodes, storage)

• Submit and manage jobs with SLURM (sinteractive, sbatch)

• Set up SSH port forwarding to access Jupyter notebooks on remote compute nodes