Virtual Environments

Virtual Environments#

When developing software, particularly in data science and quantitative finance, managing dependencies and ensuring consistent environments across different systems are crucial. A common tool that you will encounter are Conda environments, often using environment.yml files, or sometimes paired with pip and its requirements.txt files. This short note will explain the importance of these tools and files in creating reproducible and reliable software environments.

Examples: Streamlit Finance Dashboards#

These two examples demonstrate examples of end products that might inspire you. These are very simple examples that are easy to construct, but they are here to give you ideas for what you might want to build.

Reproducing these example is made easy with the use of software environments.

Example 1: Streamlit Finance Dashboard#

This app is a simple example of using Streamlit to create a financial data web app. This demo use streamlit, pandas and yfinance modules.

Allows you to select one of the 500 companies that compose the S&P 500 and display a updated chart of adjusted closing prices, as well as add a pair of moving averages.

Sample Animation

Clone the repository located here: jmbejara/streamlit_finance_chart
Navigate the command line to the source directory. Or, open the cloned repo in VS Code and then open the VS Code terminal.
In the terminal, use the following commands to create and install the associated environment.

conda create -n streamlit python=3.12
conda activate streamlit
pip install -r requirements.txt

You can then run the dashboard with

streamlit run app.py

After this, you can remove the conda environment if you like: conda remove --name streamlit --all

Example 2: Another Streamlit Dashboard, except that it requires API keys for data service#

FinStockDash is a web application that allows users to analyze stock data using various financial ratios, balance sheet, income statement, and other financial metrics. User can enter a stock ticker symbol to retrieve relevant financial information, view stock prices over time, and generate visualizations to gather insights on the company performance. The application makes use of APIs provided by Financial Modeling Prep and Alpha Vantage to retrieve real-time stock data.

Demo

This example requires API keys for data service. API keys are an example of a secret. Secrets are a type of information that should not be shared with others. In this example, the secrets are stored in a file called .env. We’ll learn more about this in today’s lecture.

Clone the repository located here: jmbejara/FinStockDash
Navigate the command line to the source directory. Or, open the cloned repo in VS Code and then open the VS Code terminal.
In the terminal, use the following commands to create and install the associated environment.

To run the app locally, follow these steps:

Clone the repository:

$ git clone https://github.com/abeltavares/financial_dashboard_app.git

Create and activate a virtual environment:

$ python3 -m venv .venv
$ source .venv/bin/activate

Install the required libraries:
```
$ pip install -r requirements.txt
```
Create a .env file with the proper API keys. Use the .env.example file as a template.
Run the app:
```
$ streamlit run app.py
```

You can deactivate the environment with

deactivate

You can delete the virtual environment with

rm -rf .venv

or on Windows command prompt:

rmdir .venv

(or rmdir /S /Q .venv to remove the directory and all its contents)

Understanding Dependency Management#

Dependencies are external libraries or packages that your project needs to function correctly. In data science, examples include NumPy for numerical operations, pandas for data manipulation, and Matplotlib for plotting.

Why Manage Dependencies?

Consistency: Ensures that all developers and users work with the same versions of libraries, avoiding the “it works on my machine” problem.
Isolation: Protects system-level Python and prevents conflicts between project dependencies.

Pip and requirements.txt#

pip is the standard package manager for Python, allowing for the installation of packages from the Python Package Index (PyPI). A requirements.txt file is

a text file listing the dependencies needed for a project.
Each line in requirements.txt represents a package and its version.
Use pip install -r requirements.txt to install all dependencies at once.

For example, these are the contents of the requirements.txt file used to generate this textbook:

altair==5.1.2
beautifulsoup4==4.12.2
colorama
doit==0.36.0
ipython==8.17.2
jupyter==1.0.0
jupyterlab==4.5.7
jupyter-book==0.15.1
linearmodels==5.3
matplotlib==3.8.1
notebook==7.5.6
numpy==1.26.0
openpyxl==3.1.2
pandas==2.1.2
pandas-datareader==0.10.0
pandas-market-calendars==4.3.1
plotly==5.18.0
plotnine==0.12.4
polars==0.19.12
python-decouple==3.8
python-dotenv==1.0.0
pytest==7.4.3
pyxlsb==1.0.10
quantstats
requests==2.33.0
scikit-learn==1.5.0
scipy==1.11.3
seaborn==0.13.0
statsmodels==0.14.0
streamlit==1.54.0
vega_datasets==0.9.0
weightedstats==0.4.1
wrds==3.1.6
xbbg==0.7.7
xlrd==2.0.1
xlsxwriter
yfinance
zstandard==0.22.0

This allows us to install all of the needed packages at once. However, this does not provide isolation from other projects. This is where a virtual environment becomes useful.

What is a virtual environment?#

A virtual environment is an isolated workspace that allows you to manage separate sets of dependencies and Python versions for different projects, preventing conflicts and maintaining a clean and manageable development setup. Some examples of software within the Python ecosystem that provide such environments are venv, virtualenv, or conda environments. Here, I’ll focus on conda environments.

Conda environments#

conda is a package manager that handles Python packages and also binaries for other languages. It is particularly useful in data science due to its ability to manage complex dependencies and environments. A major advantage of conda, at least in the past, was its ability to handle dependencies on non-Python packages. conda as a package manager can install non-Python dependencies.

Conda’s environment.yml files

Besides functioning as a package manager to handle dependencies, it also provides virtual environments. An advantage of conda environments is that they can easily manage different versions of Python as well as Python packages. This must be done in a more manual fashion with something like venv.

The description of the software environment is defined by the environment.yml file. It is

a YAML file that specifies not just Python packages but also the Python version and other binaries.
conda env create -f environment.yml creates an isolated environment with specified dependencies.
conda activate <env_name> to activate the environment.

The following is an example of an environment.yml file:

name: finm
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.12
  - altair>=5.1.2
  - beautifulsoup4>=4.12.2
  - doit>=0.36.0
  - ipython>=8.17.2
  - jupyter>=1.0.0
  - jupyterlab>=4.0.11
  - linearmodels>=5.3
  - matplotlib>=3.8.1
  - notebook>=7.0.6
  - numpy>=1.26.0
  - openpyxl>=3.1.2
  - pandas>=2.1.2
  - pandas-datareader>=0.10.0
  - pandas-market-calendars>=4.3.1
  - plotly>=5.18.0
  - plotnine>=0.12.4
  - pytest>=7.4.3
  - python-decouple>=3.8
  - python-dotenv>=1.0.0
  - pyxlsb>=1.0.10
  - requests>=2.31.0
  - scikit-learn>=1.3.2
  - scipy>=1.11.3
  - seaborn>=0.13.0
  - statsmodels>=0.14.0
  - streamlit>=1.28.1
  - vega_datasets>=0.9.0
  - xbbg>=0.7.7
  - xlrd>=2.0.1
  - zstandard>=0.22.0
  - pip
  - pip:
    - polars-lts-cpu # This is for compatibility with older CPUs, like the one the autograder uses.
    - wrds

Benefits of conda Environments

Cross-Language: Manages dependencies for languages other than Python.
System Independence: Works across different operating systems.
Environment Cloning: Allows for easy replication of environments.

Performance Issues with Conda#

For years, conda was known for persistent performance issues, such as slow package installation, dependency resolution taking a long time, and a sluggish environment creation or update process. This was primarily due to its comprehensive package management system that handles both Python and non-Python packages. The Anaconda team has since done a lot of work here: recent versions of conda ship with a much faster dependency solver by default, which dramatically speeds up environment creation and updates. For more information, see here:

My recommendation#

In my experience, it has been easier to work with a combination of conda and pip. I have found that many others have gone this route as well. This involves creating a new blank conda environment for each project and then using pip install -r requirements.txt into the conda environment.

For this course, use conda + pip so that everyone’s setup matches. This is what the homework instructions assume, and it keeps us all on the same page when we debug environment issues together.

A note on newer tools: uv and pixi#

The Python tooling landscape is moving fast, and two newer tools are worth knowing about now—even though we’ll stick with conda + pip for the homework in this course. We’ll come back to these in a later week and work through hands-on examples.

uv is an extremely fast (often 10–100x faster) drop-in replacement for pip and venv, written in Rust. The key point for now: it reads the very same requirements.txt you already have. For example, uv pip install -r requirements.txt does what pip install -r requirements.txt does, just much faster. See https://docs.astral.sh/uv/.
pixi is a modern, conda-compatible environment manager (also written in Rust). It pulls packages from conda-forge—like conda—but with a faster solver and a lockfile-first workflow that records exact versions for fully reproducible environments. See https://pixi.sh/.

The thing to take away today is that your requirements.txt (and, for conda-style packages, your environment.yml) is the portable interchange format. The same file works whether you install it with pip, uv, or conda. So the habits you build now carry over directly when we explore these newer tools later.

For now, though: stick with conda + pip. Just know that uv and pixi exist—we’ll go deeper on uv in particular, with a smaller example of pixi, in a later week.

If you’d like a preview, you can see all four approaches—conda only, conda + pip, uv, and pixi—running the same script side by side in the software_environments/ directory of the in-class examples repository, which we’ll use throughout the course.

Cheat sheet#

A short tutorial can be found here: https://edcarp.github.io/introduction-to-conda-for-data-scientists/02-working-with-environments/index.html
The official conda documentation can be found here: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment

Working with pip requirements

conda allows for a lot of flexibility, but can often be slow. pip, however, is fast for what it does. You can install the requirements for this project using the requirements.txt file specified here. Do this with the following command:

pip install -r requirements.txt

Working with conda environments

The dependencies used in this environment (along with many other environments commonly used in data science) are stored in the conda environment called finm which is saved in the file called environment.yml. To create the environment from the file (as a prerequisite to loading the environment), use the following command:

conda env create -f environment.yml

Now, to load the environment, use

conda activate finm

Note that an environment file can be created with the following command:

conda env export > environment.yml

However, it’s often preferable to create an environment file manually, as was done with the file in this project.

Also, these dependencies are also saved in requirements.txt for those that would rather use pip. Also, GitHub actions work better with pip, so it’s nice to also have the dependencies listed here. This file is created with the following command:

pip freeze > requirements.txt

Other helpful conda commands

Create conda environment from file: conda env create -f environment.yml
Activate environment for this project: conda activate finm
Remove conda environment: conda remove --name finm --all
Create blank conda environment: conda create --name finm --no-default-packages
Create blank conda environment with different version of Python: conda create --name finm --no-default-packages python Note that the addition of “python” will install the most up-to-date version of Python. Without this, it may use the system version of Python, which will likely have some packages installed already.