Python

Getting Started with Python in ExCL with best practice recommendations.

This page covers a few recommendations and tips for getting started with Python in ExCL following best practices for packaging python projects and using virtual environments. There are many different ways to structure and package python projects and various tools that work with python, so this page is not meant to be comprehensive but to provide a few recommendations for getting started.

Python Virtual Environments Using uv

Aaron recommends using uv to manage your Python virtual environments. This approach works best to avoid edge cases found with heterogeneous clusters like ExCL. That is because UV installs in your home directory without any special privileges, and it allows you to install and create Python virtual environments very efficiently (think 10-100x faster) using any version of Python.

Here is how I use it…

I use a fish function that I wrote and provide as part of the default fish shell config on ExCL. This function is pvenv which stands for Python virtual environment. This function updates uv—something that I would otherwise forget to do—and then it either sources an existing .venv environment or creates a new Python environment stored in .venv and activates it. Once the environment is active, you can then use uv pip to install new packages.

If you are not on ExCL, here is the fish function for reference.

pvenv.fish:

function pvenv --wraps='uv venv --seed' --description 'Create and activate a python virtual environment in .venv with updated pip and prompt set to the folder\'s name'
   uv self update
   if test -e .venv/bin/activate.fish
      echo Using existing `.venv`.
      source .venv/bin/activate.fish
   else
      echo Creating new `.venv`.
      # python3 -m venv --upgrade-deps --prompt (basename $PWD) .venv $argv; and source .venv/bin/activate.fish;
      uv venv --seed $argv; and source .venv/bin/activate.fish;
   end
end

How to Install uv

Follow the installation guide at Installation | uv.

Using UV to create a python virtual environment with a specific version of python.

When a specific version of python is required, uv can be used to create a virtual environment with the specific version of python.

uv venv --python <version>

For example:

uv venv --python 3.11

Use the command below to see the available python versions.

uv python list

See astral-sh/uv - python management and uv docs - installing a specific version for details.

Python Virtual Environments with venv

Using virtual environments is the recommended way to isolate Python dependencies and ensure compatibility across different projects. Virtual environments prevent conflicts between packages required by different projects and simplify dependency management. The goal with isolated, project specific python environments is to avoid the situation found in https://xkcd.com/1987/.

xkcd 1987 - Python Environment

If you are a using the fish shell, the simple function show below is a wrapper around venv to activate a python virtual environment if one already exists in .venv in the current directory or create a new virtual environment and activate it if one does not already exist.

function pvenv --wraps='python3 -m venv --upgrade-deps venv' --description 'Create and activate a python virtual environment in .venv with updated pip and prompt set to the folder\'s name'
   if test -e .venv/bin/activate.fish
      echo Using existing `.venv`.
      source .venv/bin/activate.fish
   else
      echo Creating new `.venv`.
      python3 -m venv --upgrade-deps --prompt (basename $PWD) .venv $argv; and source .venv/bin/activate.fish;
   end
end

This pvenv function is already configured system wide for fish on ExCL systems.

To create the virtual environment without using the wrapper function is also easy.

In bash:

python3 -m venv --upgrade-deps --prompt $(basename $PWD) .venv
source .venv/bin/activate

In fish:

python3 -m venv --upgrade-deps --prompt (basename $PWD) .venv
source .venv/bin/activate.fish

Here is the usage of venv which explains what the various flags do. From venv — Creation of virtual environments — Python 3.13.1 documentation.

usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
            [--without-scm-ignore-files]
            ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.

positional arguments:
  ENV_DIR               A directory to create the environment in.

options:
  -h, --help            show this help message and exit
  --system-site-packages
                        Give the virtual environment access to the system
                        site-packages dir.
  --symlinks            Try to use symlinks rather than copies, when
                        symlinks are not the default for the platform.
  --copies              Try to use copies rather than symlinks, even when
                        symlinks are the default for the platform.
  --clear               Delete the contents of the environment directory
                        if it already exists, before environment creation.
  --upgrade             Upgrade the environment directory to use this
                        version of Python, assuming Python has been
                        upgraded in-place.
  --without-pip         Skips installing or upgrading pip in the virtual
                        environment (pip is bootstrapped by default)
  --prompt PROMPT       Provides an alternative prompt prefix for this
                        environment.
  --upgrade-deps        Upgrade core dependencies (pip) to the latest
                        version in PyPI
  --without-scm-ignore-files
                        Skips adding SCM ignore files to the environment
                        directory (Git is supported by default).

Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.

The virtual environment can be exited with deactivate.

Creating a Python Project in using the Hatch build system with CI support

Python Project Template provides a template for creating a python project using the hatch build system with CI support using ORNL's GitLab instance, complete with development documentation, linting, commit hooks, and editor configuration.

Steps to use the template:

1. Fork the repository.

2. Run setup_template.sh to set up the template for the new project.

3. Remove setup_template.sh

See Python Project Template Documentation for details on the template.