The ChartBook Catalog: Data Dependencies Across Repositories

The ChartBook Catalog: Data Dependencies Across Repositories#

Objectives of This Chapter

Understand the difference between a ChartBook pipeline and a ChartBook catalog, and why a catalog is the right tool for sharing data between repositories.
Upgrade to ChartBook 0.1.x and understand what changed in this release (a breaking manifest redesign, catalog auto-discovery, and a simpler install).
Set up a personal catalog on your own machine, register cloned pipeline repos in it, and load their data by name with data.load(...).
Apply this to the Midterm Report: build your figure on top of data produced by the FTSFR pipeline repos instead of re-writing data pulls from scratch.

The Problem: Dependencies Between Repositories#

By now you have built several pipelines: repositories where doit pulls raw data, transforms it, and produces registered dataframes, charts, and notebook reports. Each pipeline is self-contained, which is exactly what makes it reproducible.

But real analytical work is rarely one repository deep. Your midterm-report figure might need consolidated CRSP Treasury data, or the Fed’s published yield curve, or Markit CDS spreads. Some other repository has already solved that data problem, carefully. Your options:

Copy the pull code into your repo. Fast today, painful forever: the copy drifts out of date, bugs get fixed in one place but not the other, and every project pays the full data-cleaning cost again.
Copy the output file by hand. Now nobody knows how that parquet was made. This violates everything this course stands for.
Declare a dependency on the other repository’s registered output, and resolve it by name at load time.

Option 3 is what the ChartBook catalog does. A pipeline registers its dataframes in its chartbook.toml; a catalog aggregates many pipelines and gives every dataset a stable, loadable address: pipeline + dataframe. Downstream code then says what it needs, not where it lives on your disk:

from chartbook import data

df = data.load(pipeline="ftsfr/fed_yield_curve", dataframe="fed_yield_curve", format="pandas")

This is the same discipline that package managers brought to code dependencies (import finm instead of copy-pasting functions), applied to data.

ChartBook 0.1.x: What Just Changed#

This week ChartBook shipped its biggest release so far (0.1.0 on July 22, 0.1.1 on July 23). Upgrade before class:

pip install --upgrade chartbook

Highlights of the release, from the changelog:

Single install. pip install chartbook now installs everything: data loading, plotting, the Sphinx documentation toolchain, and the chartbook init scaffolder. The old [data], [plotting], [sphinx], and [all] extras are deprecated no-op aliases.
Breaking: manifest format v2. The chartbook.toml format was redesigned, and v1 files are no longer supported. The old [config], [site], and [pipeline] sections collapse into one [project] table; every field is now optional with sensible defaults (the minimal valid pipeline manifest is an empty file); and entity fields are de-prefixed (chart_name → name, path_to_parquet_data → path, topic_tags → tags, and so on). A migration script in the chartbook repo (python scripts/migrate_toml_v2.py <project>) rewrites a v1 file for you.
Scoped pipeline identity. Pipelines are now canonically identified as scope/name — for example ftsfr/crsp_treasury — derived from the repo’s git remote and directory name. Bare names still work when unambiguous.
Catalog auto-discovery. A catalog can declare its members as glob patterns (members = ["../ftsfr_repos/*"]) so that new clones join automatically, with no bookkeeping.

Version numbers are a contract. This release breaks old manifests, and the version says so: under semantic versioning, a 0.x project signals breaking changes by bumping the minor version, 0.0.21 → 0.1.0. The changelog records what each version changed and why, following the Keep a Changelog convention. When we discuss packaging this week, we’ll walk through how this release was actually cut and published to PyPI.

Pipelines vs. Catalogs#

Both are described by a chartbook.toml manifest. The difference is one table:

A pipeline manifest describes one project: its [project] metadata and its registered [dataframes], [charts], and [notebooks].
A catalog manifest has a [pipelines] table — a registry of member pipelines. That table’s presence is what makes it a catalog.

The FTSFR organization is the production-scale example: roughly twenty pipeline repos (Treasury data, CDS returns, options, FX, and more), each registering its own dataframes, plus one catalog repo that pins them together:

[project]
type = "catalog"
name = "ChartBook Catalog"

[pipelines."ftsfr/crsp_treasury"]
path = "../crsp_treasury"

[pipelines."ftsfr/fed_yield_curve"]
path = "../fed_yield_curve"

# ... one entry per member pipeline

chartbook build inside the catalog directory renders one combined site: chart and dataframe listings across all members, per-pipeline pages, and a diagnostics page showing metadata completeness. Multiple pipelines, one catalog.

Setting Up Your Own Catalog#

Step 1: Clone the pipelines you need#

Make a dedicated folder and clone the FTSFR repos you want to draw from (see the table below for which ones you can actually run):

mkdir -p ~/GitRepositories/ftsfr_repos
cd ~/GitRepositories/ftsfr_repos
git clone https://github.com/ftsfr/fed_yield_curve
git clone https://github.com/ftsfr/ken_french_data_library
git clone https://github.com/ftsfr/crsp_treasury

Step 2: Register them in your global catalog#

ChartBook keeps a ready-made personal catalog at ~/.chartbook/chartbook.toml, with its own command group:

chartbook catalog init                          # create it (first time only)
chartbook catalog add ~/GitRepositories/ftsfr_repos/*   # register pipelines (globs work)
chartbook catalog build                         # build the combined site into ~/.chartbook/docs/
chartbook catalog browse                        # open that site in your browser

catalog add derives each scoped ID from the repo’s git remote (ftsfr/crsp_treasury from github.com/ftsfr/crsp_treasury) and skips duplicates. Alternatively, skip per-repo bookkeeping entirely with auto-discovery — edit ~/.chartbook/chartbook.toml to read:

[pipelines]
members = ["~/GitRepositories/ftsfr_repos/*"]

and every pipeline you ever clone into that folder joins the catalog automatically. chartbook catalog disable <id> / enable <id> switch members off and on without deleting anything.

Check what’s registered:

chartbook ls              # tree of every pipeline, dataframe, and chart
chartbook ls dataframes   # flat list of dataframes

Step 3: Run the pipeline once#

Registering a pipeline tells the catalog where its outputs will be; it does not produce them. Inside each cloned repo, install its requirements and run it:

cd ~/GitRepositories/ftsfr_repos/fed_yield_curve
pip install -r requirements.txt
doit

For the WRDS-based repos you will need your WRDS credentials in a .env file, exactly as in HW 3.

Step 4: Load data by name#

Once a pipeline has run, its dataframes are loadable from anywhere on your machine:

from chartbook import data

# default: a polars LazyFrame
lf = data.load(pipeline="ftsfr/fed_yield_curve", dataframe="fed_yield_curve")

# or ask for pandas
df = data.load(pipeline="ftsfr/fed_yield_curve", dataframe="fed_yield_curve", format="pandas")

# where is the underlying parquet?
path = data.get_data_path(pipeline="ftsfr/fed_yield_curve", dataframe="fed_yield_curve")

Pipeline references accept three forms: a bare name ("fed_yield_curve", fine when unambiguous), the canonical scoped name ("ftsfr/fed_yield_curve"), or the repository URL. The same lookups exist on the command line:

chartbook data get-path --pipeline ftsfr/fed_yield_curve --dataframe fed_yield_curve
chartbook data get-docs --pipeline ftsfr/fed_yield_curve --dataframe fed_yield_curve

Using the Catalog for Your Midterm Report Contribution#

This is the workflow for your Midterm Report figure: rather than writing a new data pull from scratch, build on a dataset an FTSFR pipeline already produces. Clone the repo, register it, run it, then data.load(...) inside your contribution’s code in the midterm-report repo. Your contribution stays reproducible (the data’s provenance is a registered, documented pipeline), and the report becomes a downstream consumer of maintained data rather than a pile of one-off scripts.

Which FTSFR repos can you use?#

Not every FTSFR pipeline is runnable by students: some require a Bloomberg Terminal subscription, which this course doesn’t provide. The repos below rely only on free public data or on WRDS (which you have access to through the course), so you can run them end to end:

Repo	Data	Access
`ftsfr/fed_yield_curve`	The Fed’s published (GSW) Treasury yield curve	Public
`ftsfr/ken_french_data_library`	Ken French Data Library factor and portfolio returns	Public
`ftsfr/he_kelly_manela`	He-Kelly-Manela intermediary asset pricing factors	Public
`ftsfr/nyu_call_report`	NYU-hosted bank Call Report data	Public
`ftsfr/corp_bond_returns`	Corporate bond returns from OpenBondAssetPricing.com	Public
`ftsfr/crsp_treasury`	CRSP daily Treasury data, plus Fed yield curve and Treasury auction stats	WRDS + public
`ftsfr/wrds_crsp_compustat`	CRSP/Compustat equity data	WRDS
`ftsfr/wrds_bank_premium`	WRDS bank regulatory data	WRDS
`ftsfr/options`	OptionMetrics equity options data	WRDS
`ftsfr/cds_returns`	Markit CDS spreads and returns	WRDS
`ftsfr/cds_bond_basis`	WRDS bond returns plus Markit CDS	WRDS + public

The remaining pipelines (basis_tips_treas, basis_treas_sf, basis_treas_swap, cip, commodities, foreign_exchange, sovereign_bonds) require Bloomberg Terminal data and are off the menu for this exercise, though you’re welcome to read their code.

Pick the pipeline closest to your claimed midterm-report topic. Where possible, choose one that also relates to your final-project paper, so the effort compounds.