To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated.

Sample acknowledgment:

This research used resources of the Experimental Computing Laboratory (ExCL) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725

You may use any variation on this theme, calling out specific simulations or portions of the research that used ExCL resources, or citing specific resources used.

However, the crucial elements to include are:

• The spelled out center name (it's okay to include the acronym, too): Experimental Computing Laboratory (ExCL)

• Office of Science and U.S. Department of Energy

• Contract No. DE-AC05-00OR22725

Additionally, when you add the paper to Resolution, please add “Experimental Computing Laboratory” to Research Centers and Institutes under Funding and Facilities as show in this image.

We appreciate your conscientiousness in this matter. Acknowledgment and pre-publication notification helps ExCL communicate the importance of its role in science to our sponsors and stakeholders, helping assure the continued availability of this valuable resource.

Thank you Chen Zhang for the presentation materials to learn about and get started with Marimo. Marimo works well in ExCL and can be set up to work with the Ollama instance running in ExCL to enable the AI features.

A Marimo server running on a worker node can be accessed either with VS Code’s automated port forwarding or via FoxyProxy

Quick Start

Very quick start using uv and FoxyProxy.

uv tool run marimo edit test.py --host $hostname --headless

Then copy and paste the provided URL into a browser with FoxyProxy set up.

Presentation

Description

EMU-Chick System is composed of 8x nodes that are connected via RapidIO Interconnect.

Each node has:

The Experimental Computing Laboratory is a Advanced Computing Systems Research project directed by Jeffrey Vetter. Support staff include:

• - systems engineer

• - software engineer

Contact for assistance.

Two Nvidia H100s are now available on hudson.ftpn.ornl.gov. From Nvidia documentation:

The NVIDIA H100 NVL card is a dual-slot 10.5 inch PCI Express Gen5 card based on the NVIDIA Hopper™ architecture. It uses a passive heat sink for cooling, which requires system airflow to operate the card properly within its thermal limits. The NVIDIA H100 NVL operates unconstrained up to its maximum thermal design power (TDP) level of 400 W to accelerate applications that require the fastest computational speed and highest data throughput. The NVIDIA H100 NVL debuts the world’s highest PCIe card memory bandwidth of nearly 4,000 gigabytes per second (GBps)

Basic validation has been done via running the nvidia samples nbody program on both devices:

The GPUs are available to the same UIDs as are using the A100s on milan0. If nvidia-smi does not work for you, you don't have the proper group memberships -- please send email to [email protected] and we will fix it. nvhpc is installed as a module as it is on other systems.

See to learn more about Julia.

Use module load julia to load the Julia tooling on an ExCL system.

Julia VSCode Extension in ExCL

Since Julia is install and loaded as a module, the has trouble finding the Julia executable needed to run properly. Therefore to use the extension on ExCL worker nodes via

Currently has a U250 installed with a custom application deployed which requires an older linux kernel.

Lewis is configured with kernel 5.15.0.

Hold set with:

To remove hold:

ExCL reserves the first Tuesday of every month for systems maintenance. This may result in complete inaccessibility during business hours. Every effort will be made to minimize the scope, duration, and effect of maintenance activities.

If an outage will affect urgent projects (i.e., with impending deadlines) please email [email protected] as soon as possible.

I can't log in

There are two most likely sources of this problem

Password won't work

The most frequent cause is having your visitor (non-ORNL internal password) wrong, or having had it expire. See https://xcams.ornl.gov to address this. If you are ORNL staff, a frequent cause is a failure to keep your internal ORNL systems password up to date (UCAMS) or have missed required training. ExCL makes the same check that any ORNL system makes as to whether a password is valid or an account exists (you will not be able to differentiate the two errors based on the login failure). This will look like

Too many login attempts from a given IP address

ExCL limits logins to five consecutive failures within a short period of time. After that limit exceeded, login attempts from your IP address will be blocked. This might look like

To have this addressed, report your IP address to [email protected]. If you are on an ORNL network, you can use the usual native tools on your system to find your IP address. If you are at home and on a network using NAT (as most home networks do) use to determine your public IPv4 address when external to the lab. Note that this will not report the correct address if you are on an ORNL (workstations or visitor) network.

I can’t clone my git repo

The recommended approach for accessing git repositories in ExCL is to use the SSH protocol instead of the HTTPS protocol for private repositories and either protocol for public repositories. However, both approaches will work with the proper proxies, keys, applications passwords, and password managers in place.

To use the SSH protocol you must first setup SSH keys to the git website (i.e. GitLab, GitHub, and Bitbucket). See for details for how to do this for code.ornl.gov. The other Git Clouds have similar methods to add SSH keys to your profile.

Since the worker nodes are behind a proxy. You must setup an SSH jump host in your .ssh/config to access Git SSH servers. See to verify that you have set up the proper lines in your SSH Config.

I need a newer version of pip

See for instructions on how to set up a Python virtual environment with the latest version of pip.

I need a newer version of Python

See for instructions on how to use UV to set up a Python virtual environment with a specific python version.

I can't access git/web/whatever from within Docker

Docker does not pick up user environment variables when a container is started, nor does it pick up variables from the Docker daemon. These environment variables are required for Docker to access services outside of the ExCL environment (via a squid proxy).

This can be done by creating ~/.docker/config.json with the contents:

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

Once you have access you have a couple of options.

• login.excl.ornl.gov runs an SSH Server and you can connect to the login node with ssh login.excl.ornl.gov.

• There is a limited number of licenses available. Thinlinc (Xfce Desktop) can be accessed at for HTML5 services, and ThinLinc clients can use login.excl.ornl.gov as their destination. ThinLinc clients can be downloaded without cost from . ThinLinc provides much better performance than tunneling X over SSH. A common strategy is to access login.excl.ornl.gov via ThinLinc and then use X11 forwarding to access GUIs running on other nodes.

Notes:

• Using an SSH key instead of a password to connect to ExCL is highly recommended. See . SSH keys are more secure than passwords, and you are less likely to accidentally get banned from multiple incorrect login attempts when using SSH Keys to authenticate. If you get blocked, you can send a help ticket to with your IP address to get removed from the block list.

• If you use a passphrase with your SSH key (recommended for security), you should also set up an SSH Agent to load the SSH key. An SSH Agent allows you to enter your passphrase once for the session without needing to enter your passphrase many times. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms; see .

• It is recommended to use a terminal multiplexer like

Add SSH Public Key to ExCL’s Authorized Keys

You can manually copy the key if already on ExCL. For example

Or you can you ssh-copy-id to copy your local systems key to ExCL.

Description

The Qualcomm® Snapdragon™ 855 Mobile Hardware Development Kit (HDK) is a highly integrated and optimized Android development platform.

Accessing this system:

• Qualcomm board is connected to an HPZ820 workstation (McMurdo) or to an HP Z4 workstation (Clark) through USB

• Development Environment: Android SDK/NDK

• Login to mcmurdo or clark

  • $ ssh –Y mcmurdo

• Setup Android platform tools and development environment

  • $ source /home/nqx/setup_android.source

• Make sure you have a functioning environment

  • adb kill-server

  • adb start-server

• Run Hello-world on ARM cores

  • $ git clone

  • $ make compile push run

• Run OpenCL example on GPU

  • $ git clone

  • Run Sobel edge detection

Other Details

The snapdragon SDK uses python 2.7; you may need to explicitly specify python2 in your environment.

Access

Access will be granted per request (as this cannot be used as a shared resource).

Useful Links

1. Android Studio:

2. Qualcomm HDK:

3. Qualcomm Neural Processor SDK:

Images

Description

This system is generally identical to the nodes (AC922 model 8335_GTW) in the ORNL OLCF Summit system. This system consists of

• 2 POWER9 (2.2 pvr 004e 1202) cpus, each with 22 cores and 4 threads per core.

• 6 Tesla V100-SXM2-16GB GPUs

• 606GiB memory

• automounted home directory (on group NFS server)

Contact

• [email protected]

Usage

As currently configured this system is usable using conventional ssh logins (from login.excl.ornl.gov), with automounted home directories. GPU access is currently cooperative; a scheduling mechanism and scheduled access is in design.

The software is as delivered by the vendor, and may not be satisfactory in all respects as of this writing. The intent is to provision a system that is as similar in all respects to Summit, but some progress is required to get there. This is to be considered an early access machine.

Please send assistance requests to [email protected].

Installed Compilers

Please see

GPU Performance

This system is still being refined with respect to cooling. As of today, rather than running at the fully capable 300 watts per GPU, GPU usage has been limited to 250 watts to prevent overheating. As cooling is improved, this will be changed back to 300 watts with dynamic power reduction (with notification) as required to protect the equipment.

It is worth noting that this system had to be pushed quite hard (six independent nbody problems, plus CPU stressors on all but 8 threads) to trigger high temperature conditions. These limits may not be encountered in actual use.

Performance Information

GPU performance information can be viewed at

Request access by emailing [email protected].

Other Resources

• IBM 8335-GTW documentation:

This is the user documentation repository for the Experimental Computing Laboratory (ExCL) at Oak Ridge National Laboratory.

This site is undergoing development; systems and processes will be documented here as the documentation is created.

See the index on the left of this page for further detail.

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated. See Acknowledgment for details.

See Requesting access for information on how to request access to the system.

How to Login

See for more details.

• Shell login: ssh login.excl.ornl.gov

• ThinLinc Session:

Getting Assistance

Please send an email request to for assistance. This initiates a service ticket and dispatches it to ExCL staff. Please include the following:

• System name (i.e. faraday, login).

• Software name (if causing problems or if requesting an installation)

• Your UID (login name). Do not send a password.

ExCL Cheat Sheet

The MI300A system (host name faraday) is available for ExCL users. As usual you have to log in through the login node.

Make sure that you

to set up all of the environment needed.

A very light test program is available via git at https://github.com/jungwonkim/amd-toy. This is a good way to ensure your environment is set up correctly.

All tests should return err[0]. If they do not, then it is likely that you do not have render group permissions

To check, run the groups command (on faraday) and see if you are in the render group.

If you are not, contact , and we’ll get you in.

System Information

• Supermicro AS -4145GH-TNMR

  • No configuration options wrt memory or other addons.

• 4 APU (Accelerated Processing Unit) (combined CPU, GPU and HBM3 memory)

Documentation

• Available models:

• Datasheet on Faraday:

• Hardware documentation:

Images

Roadmap for Setup

If you are new to remote development on ExCL here is a roadmap to follow to set important settings and to get familiar with remote Linux development.

→ →

About ExCl User Documentation

Documentation published to ExCL users is available in our . Users are encouraged to contribute by improving the material or providing user-created tutorials to share with the community.

These card are currently installed on Secretariat and Affirmed, but will eventually be moved to take advantage of GPUs installed elsewhere.

Software installation

All DOCA, embedded and BSP software was updated in September 2023, using the following:

Link: Website: Documentation: GitHub:

There is an Open WebUI server running on ExCL for developing and testing LLM models created with . In order to use the website you must first , then the above link will work. When you first access the page, you will be prompted to create a new account. This account is a unique account for this instance of Open WebUI and is not tied to anything else. After creating an account, send a message to Aaron Young or

See the Quick-Start guides to get going:

• GitLab-CI in ExCL Quick-start Guide.

• GitHub-CI in ExCL Quick-start Guide.

Deprecated: See documentation at https://code.ornl.gov/excl-devops/documentation/-/tree/master/devops.

These 2U servers are highly capable large memory servers, except that they have limited PCIe4 slots for expansion.

• HPE ProLiant DL385 Gen10 Plus chassis

• 2 AMD EPYC 7742 64-Core Processors

  • configured with two threads per core, so presents as 256 cores

  • this can be altered per request

• 1 TB physical memory

  • 16 DDR4 Synchronous Registered (Buffered) 3200 MHz 64 GiB DIMMS

• 2 HP EG001200JWJNQ 1.2 TB SAS 10500 RPM Disks

  • one is system disk, one available for research use

• 4 MO003200KWZQQ 3.2 TB NVME storage

  • available as needed

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM, and for networking/DPU research.

Status

Affirmed

Affirmed is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Justify

Justify is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Pharaoh

Pharaoh is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Secretariat

Secretariat is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

This guide goes over the recommended way to install Conda and Spack in ExCL. If you are already familiar with the Conda and Spack installation process, then these tools can be installed to their default locations. One recommendation is to store the environment.yml and spack.yaml files in your git repositories to make it easy to recreate the Conda and Spack environments required for that project. The remainder of this page goes over the installation in more detail.

Installing Conda

See for the latest installation instructions. Miniforge is the recommended version of conda since its a minimal base install which defaults to using conda-forge packages.

Follow the prompts on the installer screens. Accept the license agreements. (Optional) Specify /home/$USER/conda as the installation location. Choose if you want the installer to initialize Miniforge.

Improving Conda Environment Solver Performance

To improve the performance of the Conda environment solver, you can use the conda-libmamba-solver plugin which allows you to use libmamba, the same libsolv-powered solver used by and , directly in conda.

The quick start guide is below.

See and for more information.

Installing Spack

DevDocs

This guide goes over hosting internal to ORNL documentation using ExCL’s devdocs VM. For an example of a project which uses devdocs, see Hunter and Hunter Documentation (Source).

The documentation for hunter is built with GitLab-CI. Here are the relevant lines in .gitlab-ci.yml.

Request a DevDocs Site

If you would like to host your project’s internal documentation on ExCL, please email with the following information and we can help you get started with a DevDocs subdirectory and the DevDocs GitLab Runner.

• URL

• Project Name (This will be your DevDocs subdirectory)

pcie

Description

This system is intended for pci-based device support.

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Atipa

• Tyan Motherboard S7119GMR-06

• 192 GB memory

• Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHzIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz 2x16 cores no hyperthreading

Use

This system is used for heterogeneous accelerator exploration and FPGA Alveo/Vitis-based development.

Current VMs

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

See the Cheat Sheet for a quick summary.

Backup

User files (home directories) are stored on an ZFS-based NFS server, and are generally available to all ExCL systems (there are exceptions for operational and security reasons; if you trip over something please let [email protected] know). The /noback/<user> facility is no longer supported and is not being created for new users accounts. Files already in the /noback hierarchy will not be affected; if you would like assistance in moving these files to your home directory please let know. Space available to /noback is limited.

ExCL uses ZFS with snapshots. Zrepl () handles both automated snapshot generation and file system replication. Snapshots are taken hourly, and ExCL file systems are replicated to the back up (old FS00) fileserver. The snapshot directory name format is ~/.zfs/snapshots/zrepl_yyyymmdd_hhmmss_000 (where the hour is in UCT not Eastern Daylight/Standard Time). The use of UCT in the snapshot name is a zrepl property to enable global replication consistency, and is not modifiable. If you deleted or made a destructive modification to, say, ~/.bashrc on, say, June 11, 2024 at 3 PM, it should be available in ~/.zfs/snapshots/zrepl_20240611_185313_000/.bashrc, and in earlier snapshots.

Snapshots take space for files that have changed or been deleted. They are automatically deleted as they age, so that hourlies are kept for 48 hours, one hourly from each day is kept for 30 days, and one hourly for each 30 day period is kept for 180 days. This policy can be modified on request. Snapshots are read only; you can copy files from them back into your home directory tree to restore them.

There is currently no file purge policy. Given that ExCL researchers take care of cleaning up files that are no longer in use, no change to this policy is foreseen. Files for inactive users are archived in a non-snapshot file system. While it is our intent to continue maintaining storage for inactive users, this policy may change in the future.

Quotas

Refquotas are applied to the ZFS filesystems to avoid runaway storage usage. A refquota limit applies only to your files, not to snapshot storage. ZFS stores data in a (very fast) compressed format, so disk usage may appear to be less than you expect. Home and project subvolumes start with a refquota of 512G. Users can request higher quotas via . We can also help diagnose the cause of large storage use by providing a breakdown of file usage and helping clean up unneeded large files and snapshots.

Local Storage

In addition to shared network storage, each system has a local /scratch directory. The size will vary from system to system, and some systems may have /scratch2 in addition. A working space can be created withmkdir /scratch/$USER if one is not already present. This storage location is good for caching files on local host storage, for speeding up tasks which are storage IO bound, and performing tasks which fail on NFS storage (for example, Apptainer and embedded Linux builds). If you require more scratch storage than is available contact as on newer systems there is often additional storage available that has not been allocated. Similarly contact us if there is no /scratch or /scratch2 directory. Since there is (currently) no purging policy, please clean up after you no longer need your scratch space.

Project Storage

Shared storage space for collaborative projects is available upon request. Each project is assigned a dedicated subvolume within the ZFS filesystem, which is accessible via an automounted NFS share. The mount point for each project is located at:

Access to the project directories is restricted for security and organization. Only execute permissions are set on the /auto/projects/ directory, meaning you must know the specific project name to cd into it. You will not be able to use ls to list all available project directories.

Access Control Lists (ACLs) are used to manage permissions for project directories, allowing for flexible access configurations. By default, all members associated with a project will have read, write, and execute permissions for the files within their assigned project directory.

ExCL uses Modules to manage software environments efficiently. Modules allow users to load, unload, and switch between different software versions without modifying system paths manually. Please let us know if there is a software package you would like us to make available via a module.

Loading a Module

To load a specific software module:

Example:

This makes Python 3.9 available for use.

You can also leave off the version number to load the default version.

Example:

Listing Available Modules

To see all available modules:

Checking Loaded Modules

To view currently loaded modules:

Unloading a Module

To remove a specific module:

Example:

Switching Between Versions

To switch from one module version to another:

Example:

Resetting the Environment

To clear all loaded modules and reset to the default environment:

ExCL-Utils

The excl-utils module contains common Rust CLI utilities. Load the module to see an updated list of the installed utilities. Additional Rust CLI utilities can be requested to be included in the module. These utilities are updated to the latest versions nightly.

Ollama is deployed in ExCL as a module. To use Ollama, load the module, and then you have access to the ollama CLI interface.

Load the Ollama module with:

Ollama has a server component which stores files in its home. This server component should be launched using a service account by ExCL admin, since it provides ollama for the entire system. Ollama is already running on some of the workers in ExCL. See the output from the model load for an up-to-date list. Contact [email protected] if you would like ollama to be available on a specific system.

Ollama API

Examples of using the Ollama API can be found at .

Links

oswald00

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

Compilers are, in general, maintained from a central NFS repository, and made accessible via the module command (from Lmod). For example

If you do not load a module, you will get the default compiler as delivered by the operating system vendor (4.8.5 on some systems). If you module load gnu you will currently get 12.1.0, as it is the default. If you need, say, 10.2.0, you need to module load gnu/10.2.0. Note that documentation details with respect to compiler availability and versions will not necessarily be kept up to date; the system itself is authoritative.

Some compilers (notably xlc and the nvhpc tool chain) cannot be installed on nfs, so if they are available they will show up in a different module directory. The same module commands are used.

Additional compilers can be installed on request to . Maintaining multiple Gnu suites is straightforward, less so for other tool suites.

The EDA tools are installed on the system dragon. dragon can be access via ssh from the login node, via x11 forwarding from the login node's ThinLink, or directly via ThinLink with Foxy Proxy. See to get started with ThinLinc Setup. See for more details on logging in.

Ssh access:

ThinLinc access to login:

Hello World built with nvhpc

Load the Nvidia HPC SDK environment module

Verify the compiler path

Build the program

Run the program with MPI

Register a Runner

Runners can be registered as either a group runner or for a single repository (also know as a project runner). Group runners are made available to all the repositories in a group.

Send the following information to [email protected] and we will register the runner as a system runner.

• URL

• Registration Token

• Executor (choose shell or docker with image)

• Project Name (This can be group name or repo name)

• ExCL System

• Tag List

The method for obtaining this information differs depending on if you want to register a group runner or a single repository runner. See and sections below.

After the runner is added, you can edit the runner to change the tags and description.

Group Runner

Navigate to the group page. Click on Build → Runners. Then select New group runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

Single Repo Runner (Project Runner)

Navigate to the repo page. Click on Settings → CI/CD → Runners. Then select New project runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

List of ExCL Systems with a runner

Any system can be requested as a runner. These systems are already being used as a runner. (Updated October 2023)

• docker.ftpn.ornl.gov

• explorer.ftpn.ornl.gov

• intrepid.ftpn.ornl.gov

Using Slurm with Gitlab CI

The system slurm-gitlab-runner is setup specifically to run CI jobs that then run the execution using slurm with sbatch --wait.

For a complete example and template for how to use the Slurm with GitLab in ExCL see and .

This template includes two helper scripts, runner_watcher.sh and slurm-tee.py.

runner_watcher.sh watches the CI job and cancels the Slurm job if the CI job is canceled or times out.

slurm-tee.py watches the slurm-out.txt and slurm-err.txt files and prints their content to std-out so that the build log can be watched from the GitLab web interface. Unlike regular less --folow, slurm-tee watches the multiple files for changes and also exits once the slurm job completes.

Spack and Conda with Gitlab-CI

Groq Links

Login and Groq Cards Available

Start by logging into ExCL's login node.

From the login node, you can then login to a node with a Groq card, for example

Here is a table of the Groq cards available:

Groq card and Slurm

The recommended way to access the Groq card is to reserve it through the Slurm resource manager. Groq cards are available on machines in the groq partition. To reserve a node with a groq card for interactive use use the command.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

Non-interactive batch jobs can similarly be launched.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

or specified in the script:

Using the Groq Card

In order to use the Groq API you need to make sure you are using python 3.8 and that you add the Groq python libraries to your path. For python 3.8 you can either use the installed system python3.8 or use conda to install python3.8.

System python3.8

You need to fully quantify the path to python since Ubuntu 22.04 defaults to python3.10. This means you need to use

Then to install jupyter notebook in your home directory, you would need to do

Conda

First install miniconda by following . Then create a groq environment with

See the for more details for setting up the Conda environment.

Graphical Access to Groq Systems

See .

Jupyter Notebooks

See for more information on setting up Jupyter Notebooks within ExCL.

Getting started Resources

Useful Groq Commands

• Run regression tests to verify card functionality: tsp-regression run

• Get Groq device status: /opt/groq/runtime/site-packages/bin/tsp-ctl status

• Monitor temperature and power: /opt/groq/runtime/site-packages/bin/tsp-ctl monitor

ExCL → User Documentation → Contribute → Authoring Guide

Authoring Guide for ExCL

Perhaps you've got some how-to documents tucked away in folders that you'd like to share with the community. Or maybe you've discovered a way of doing things that would benefit other users.

You can submit your user guides for publication within the ExCL documentation site! See the contributing page for instructions.

We've assembled here the fundamental authoring guidelines for ExCL user documentation.

Document and Content Preferences

• Documents should be created using using the syntax.

• Oak Ridge National Laboratory (ORNL) uses the (CMOS) as a basic style guide.

• Define the first instance of every acronym in each document. Ensure that the long-form is not repeated after it is defined.

Pictures and Images

Screenshots and images cannot be resized using markdown. Therefore, we embed .html that is rendered when we publish the tutorial to the documentation site.

• Images and screenshots should be stored in a folderassets. Images and screenshots added from the gitbook interface are stored in .gitbook/assets, but issues seem to occur if this folder is modified externally from gitbook.

• Files should be named descriptively. For example, use names such as adding-IP-address.png instead of image03.png.

Other Considerations

• Have you redacted sensitive information from text and images?

• Have you removed information that is protected by copyright?

• Are you using a specific version of your software and have you included in the documentation?

Related Topics

• Using a for creating user content.

ExCl → User Documentation → Jupyter Quick Start

Installing Jupyter

Since there are many ways to install Jupyter using various python management tools, I will not reproduce the documentation here. The official documentation for installing Jupyter can be found at Project Jupyter | Installing Jupyter. However, I will highlight the methods of using Jupyter with UV, running Jupyter Notebooks in VS Code, and the alternative to Jupyter notebooks, Marimo | ExCL User Docs. These methods are all the methods that I typically use when working with python notebooks.

Jupyter with UV

See the UV documentation, . This documentation is well written and covers:

Jupyter kernels using virtual environments

See . Although , the following steps are still a good way to manually create and use a kernel from Jupyter.

Create a python virtual environment and activate it. Then install ipykernel and then install the kernel for use in Jupyter.

Use jupyter kernelspec list to view all the installed Jupyter kernels.

To uninstall a Jupyter kernel use uninstall.

Accessing a Jupyter Notebook Running on ExCL

A Jupyter notebook server running on ExCL can be accessed via a local web browser through port forwarding the Jupyter notebook's port. By default, this is port 8888 (or the next available port). This port might be in use if someone else is using running a notebook. You can specify the port with the --port flag when launching the Jupyter notebook. To use a different port just replace 8888 with the desired port number. In order to port forward from an internal node, you have to port forward twice, once from your machine to login.excl.ornl.gov and once again from the login node to the internal node (i.e. pcie).

Using FoxyProxy (Recommended)

You can use directly connect to the server running on a worker node. If I’m running a Jupyter Notebook on zenith2 with the command jupyter notebook, I can connect to the notebook directly using the printed message:

I hold control on Windows/Linux and click the URL to open it in my local browser.

Detailed instructions for Linux/Mac

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster. If you want to access a different system, then replace pcie with the system you intend to access.

1. Specify the ports that you intend to use. Choose a different number from the default so that you don't conflict with other users.

2. From your local machine connect to pcie using login.excl.ornl.gov as a proxy and local forward the jupyter port.

3. (Optional) Load the anaconda module if you don't have jupyter notebook installed locally.

If you ssh client is too old for proxyjump to work, you can always break up the process into another step.

1. From your local machine connect to login.excl.ornl.gov and local port forward port 8888.

2. From the login node connect to pcie and local port forward port 8888

3. Launch the Jupyter server on pcie

Detailed instructions for Windows with MobaXterm

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster.

1. From your local machine connect to login.excl.ornl.gov using MobaXterm.
2. Go to tools and click on MobaSSHTunnel. Use MobaSSHTunnel local forward port 8888.

   Click on MobaSSHTunnel

Detailed instructions for Windows with Visual Studio Code

These instructions go over how to access a Jupyter notebook running on the quad00 node in the ExCL Cluster using Visual Studio Code to handle port forwarding.

1. Open Visual Studio Code

2. Make sure you have the Remote - SSH extension installed.
3. Setup .ssh

ExCL → User Documentation → Contributing → Git Scenarios

Git Scenarios

This document includes common Git scenarios and how to deal with them.

Updating a branch with new content from the master branch

If you have been working on a development branch for a while you might like to update it with the most recent changes from the master branch. There is a simple way to include the updates to the master branch into your development branch without causing much chaos.

First, checkout your development branch. Then, perform a merge from master but add the "no fast forward" tag. This will ensure that HEAD stays with your development branch.

Resolve any conflicts and push your changes.

Configuring Git: local vs global

When you set up Git with the git config --global ... commands, you are telling your local machine that this is the set of credentials that should be used across your directories. If you have multiple projects for which you need unique credentials, you can set a particular project folder with different Git credentials by changing global to local. For example, you may contribute to projects in GitHub and GitLab. You can navigate to the local repository and set local configuration parameters. See below:

Now, the machine will use global configurations everywhere except for the /project/GitHub/ repository.

Undoing a change

Changes since your last commit

You have previously committed some files and now you've edited a file and saved your changes. However, you now decide you do not want keep the changes that you've made. How can you revert it back to the way it was at your last commit?

The git status command output provides a method for discarding changes since your last commit.

📝 Note: Before using the above commands to reverse your changes, be sure you do not want to keep them. After the commands are run, the file(s) will be overwritten and any uncommitted changes will not be recoverable.

Reverting to a previous commit

If you are working on a new feature and after a commit you realize that you have introduced a catastrophic bug, you can use git reset ac6bc6a2 (each commit has a unique identification number). This command will change where the HEAD pointer is located. For example, if you are on the master branch and have submitted three new commits, the HEAD points to your most recent commit. Using the git reset --- command will keep the information in the recent commits, but HEAD will be moved to the specified commit.

To find the unique identification number of the commits in your branch, type git log --pretty=format:"%h %s" --graph to provide a list of recent commits as well as a visual graph of changes.

Amending a commit

Let's say that you have just completed several changes, staged (added), and committed them. As you look at one file, you see a typo. You could simply fix the typo, add, and commit again, or you could use the --amend tag so that the new changes (your typo fix) can be included in your previous commit. Using this can keep your commit history uncluttered by removing commit messages such as "forgot to add a file" or "fixed a typo." Here is an example of a forgotten file amended commit:

A commit message prompt appears and you can either keep the original commit message or modify it.

Undoing a merge

Perhaps you thought you had checked out your development branch but you were, in fact, on the master branch. Then you merged a topic branch into master by mistake. How do you undo the merge?

If you just want to take a step back to before you entered the merge command, you can use git merge --abort. This is usually a safe command as long as you do not have any uncommitted changes.

If you need something a little more robust, you can use git reset --hard HEAD. This command is used to perform a "start over" in your repository. It will reset your repository to the last commit.

Collaboration Etiquette

Commit messages

When multiple people are working in the same repository, the number of commits can be anywhere between a few or several thousands depending on the size of your development team. Using clear, descriptive commit messages can help "integration managers" merge content and, perhaps more importantly, search for and find commits that have introduced a bug.

Another recommendation by the author of "Pro Git" says, "try to make your changes digestible — don’t code for a whole weekend on five different issues and then submit them all as one massive commit on Monday."

I do not want Git to track a particular file/directory

If there are files/folders in your repository that you do not want Git to track, you can add them to a .gitignore file. Here is an example .gitignore:

Works Cited

• Chacon, Scott, and Ben Straub. Pro Git: Everything You Need to Know About Git. Apress, 2nd Edition (2014).

is the most widely used container system for HPC. It is designed to execute applications at bare-metal performance while being secure, portable, and 100% reproducible. Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work.

is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Apptainer allows for more secure containers than docker without the need for root access.

The login node has install and can be accessed at . Since this node is public facing, it is the easiest to access with ThinLinc.

In addition to the login node, multiple systems including the have installed, which makes it easier to run graphical applications. To access ThinLinc you need to use as socks proxy to forward traffic to the ExCL network or port forwarding of port 22 to use the ThinLinc client.

For better keyboard shortcut support and to prevent the browser from triggering the shortcuts, I recommend installing .

This page covers a few recommendations and tips for getting started with Python in ExCL following best practices for packaging python projects and using virtual environments. There are many different ways to structure and package python projects and various tools that work with python, so this page is not meant to be comprehensive but to provide a few recommendations for getting started.

Python Virtual Environments Using

Aaron recommends using to manage your Python virtual environments. This approach works best to avoid edge cases found with heterogeneous clusters like ExCL. That is because UV installs in your home directory without any special privileges, and it allows you to install and create Python virtual environments very efficiently (think 10-100x faster) using any version of Python.

Git (code revision management system) is installed on all ExCL systems on which it makes sense. Git operates as expected, except for external access.

If you require access to external git resources, you need to do a little more.

HTTP or HTTPS access

For HTTP or HTTPS access, make sure you have the following environment variables (they should be set by default, but may not be if you have altered your environment)

→ → →

Git Workflow from the Command Line

There are many reasons one would prefer to work from the command line. Regardless of your reasons, here is how to contribute to the ExCL documentation using only command line tools.

Jump to a Section:

If you don’t want to run the runner as service then you can follow the steps posted at Adding self-hosted runners - GitHub Docs to create a self-hosted runner in ExCL.

Setup Runner as a service in ExCL

If you do want to register the runner as a service, the easiest way is to use systemd user services. To set this up follow the steps below.

Create a user systemd config which is unique to a single system.

Notes:

• If you are trying this on a system which doesn’t already have a /scratch folder the command will fail. Please send an email to to create a folder for local storage.

• If you setting up a second runner, the ln command will fail if the link already exists. Ensure that the link is a valid link pointing to scratch before continuing with these instructions.

Create a folder to store the GitHub Runner.

The steps are similar to that posted at with some changes. You will need to create one folder per machine and per repo so I recommend the following structure.

~/github-runners/<node>-<repo>

Download and Configure the Runner.

Once you create this directory and enter it, you will then download and configure the runner. The steps are reproduce below, but you should follow the instructions from the “add new self-hosted runner” page after clicking on “New self-hosted runner”.

Patch the Runner Folder for use as a User Systemd Service.

Apply this patch to modify the directory to use user systemd modules.

Enable linger for your user

Use this command to enable linger for your user.

This allows your user-level systemd services to run when you are not logged into the system and auto-start when the system is rebooted.

Note: Use loginctl disable-linger to remove linger and ls /var/lib/systemd/linger to view the users with linger set.

See for more information.

Use the svc.sh script to install and manage the runner service.

After this patch is applied the svc.sh script works as documented in . However you don’t need to specify a username since it now defaults to the current user. The commands are reproduced below.

Install service

Start service and check status.

Note: The above install adds the service to auto start on reboot. If you want to disable or enable this auto starting of the service run.

or

To stop the service run

To uninstall the service run

Creating a secure, human-in-the-loop CI pipeline for public repos

GitHub Actions the use of self-hosted runners for public repos. However, if you want to use an ExCL self-hosted runner for a public repo, you can use the following steps to create a secure CI pipeline that is triggered by an authorized user in a PR comment. This will prevent unauthorized users from running arbitrary code (e.g. attacks) automatically on ExCL systems from any PRs.

We follow the resulting workflow yaml file in the JACC.jl as an example that can be reused across repos.

1. Select authorized users: those who can trigger the pipeline and store it in a in your repo using the following format: CI_GPU_ACTORS=;user1;user2;user3; and store another secret TOKENIZER=; to be used as a delimiter (it can be any character). Users should have a strong password and 2FA enabled.

2. Trigger on issue_comment: this is the event that triggers the CI pipeline. The types: [created] ensures that the pipeline is triggered only when a new comment is made and not when an existing comment is edited.

Getting Started with Slurm

For general Slurm use and reference, see Job Submission with Slurm - CADES User Documentation, CADES Cheat Sheet, and Cheat Sheet | ExCL User Docs.

sbatch Template

Here is a recommended Slurm sbatch script template:

Using the Preemptable GPU Queue (nvidia-long)

The nvidia-long queue provides opportunistic GPU access for long-running or flexible workloads. Jobs in this queue may be preempted when higher-priority work arrives.

Preemption allows better overall cluster utilization, but your job must be written to tolerate interruption.

What “Preemptable” Means

Jobs submitted to nvidia-long:

• Run when GPUs are idle

• May be stopped at any time

• Are requeued instead of canceled only if you request it

The normal GPU queue (nvidia) is not preempted.

Important Cluster Policy

The cluster is configured with:

What this means for users

• Preemption is enabled

• Preempted jobs are eligible to be requeued

• Jobs are NOT requeueable by default