Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated.

Sample acknowledgment:

This research used resources of the Experimental Computing Laboratory (ExCL) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725

You may use any variation on this theme, calling out specific simulations or portions of the research that used ExCL resources, or citing specific resources used.

However, the crucial elements to include are:

• The spelled out center name (it's okay to include the acronym, too): Experimental Computing Laboratory (ExCL)

• Office of Science and U.S. Department of Energy

• Contract No. DE-AC05-00OR22725

Additionally, when you add the paper to Resolution, please add “Experimental Computing Laboratory” to Research Centers and Institutes under Funding and Facilities as show in this image.

We appreciate your conscientiousness in this matter. Acknowledgment and pre-publication notification helps ExCL communicate the importance of its role in science to our sponsors and stakeholders, helping assure the continued availability of this valuable resource.

Two Nvidia H100s are now available on hudson.ftpn.ornl.gov. From Nvidia documentation:

The NVIDIA H100 NVL card is a dual-slot 10.5 inch PCI Express Gen5 card based on the NVIDIA Hopper™ architecture. It uses a passive heat sink for cooling, which requires system airflow to operate the card properly within its thermal limits. The NVIDIA H100 NVL operates unconstrained up to its maximum thermal design power (TDP) level of 400 W to accelerate applications that require the fastest computational speed and highest data throughput. The NVIDIA H100 NVL debuts the world’s highest PCIe card memory bandwidth of nearly 4,000 gigabytes per second (GBps)

Basic validation has been done via running the nvidia samples nbody program on both devices:

10485760 bodies, total time for 10 iterations: 401572.656 ms
= 2738.014 billion interactions per second
= 54760.284 single-precision GFLOP/s at 20 flops per interaction

The GPUs are available to the same UIDs as are using the A100s on milan0. If nvidia-smi does not work for you, you don't have the proper group memberships -- please send email to [email protected] and we will fix it. nvhpc is installed as a module as it is on other systems.

This is the user documentation repository for the Experimental Computing Laboratory (ExCL) at Oak Ridge National Laboratory.

This site is undergoing development; systems and processes will be documented here as the documentation is created.

See the index on the left of this page for further detail.

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated. See Acknowledgment for details.

See Requesting access for information on how to request access to the system.

How to Login

See for more details.

• Shell login: ssh login.excl.ornl.gov

• ThinLinc Session:

Getting Assistance

Please send an email request to for assistance. This initiates a service ticket and dispatches it to ExCL staff. Please include the following:

• System name (i.e. faraday, login).

• Software name (if causing problems or if requesting an installation)

• Your UID (login name). Do not send a password.

• Enough detail so we can replicate the problem.

ExCL Cheat Sheet

Currently has a U250 installed with a custom application deployed which requires an older linux kernel.

Lewis is configured with kernel 5.15.0.

Hold set with:

To remove hold:

→ →

About ExCl User Documentation

Documentation published to ExCL users is available in our . Users are encouraged to contribute by improving the material or providing user-created tutorials to share with the community.

See the .

Backup

User files (home directories) are stored on an ZFS-based NFS server, and are generally available to all ExCL systems (there are exceptions for operational and security reasons; if you trip over something please let know). The /noback/<user> facility is no longer supported and is not being created for new users accounts. Files already in the /noback hierarchy will not be affected; if you would like assistance in moving these files to your home directory please let know. Space available to /noback is limited.

The Experimental Computing Laboratory is a Advanced Computing Systems Research project directed by Jeffrey Vetter. Support staff include:

• - systems engineer

• - software engineer

Contact for assistance.

Link: Open WebUI (Running on Zenith) Website: Open WebUI Documentation: 🏡 Home | Open WebUI GitHub: open-webui/open-webui: User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

There is an Open WebUI server running on ExCL for developing and testing LLM models created with Ollama. In order to use the website you must first Setup FoxyProxy, then the above link will work. When you first access the page, you will be prompted to create a new account. This account is a unique account for this instance of Open WebUI and is not tied to anything else. After creating an account, send a message to Aaron Young or [email protected] to request your account to be ungraded to an admin account.

ExCL reserves the first Tuesday of every month for systems maintenance. This may result in complete inaccessibility during business hours. Every effort will be made to minimize the scope, duration, and effect of maintenance activities.

If an outage will affect urgent projects (i.e., with impending deadlines) please email [email protected] as soon as possible.

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

See the Quick-Start guides to get going:

• GitLab-CI in ExCL Quick-start Guide.

• GitHub-CI in ExCL Quick-start Guide.

Deprecated: See documentation at https://code.ornl.gov/excl-devops/documentation/-/tree/master/devops.

Ollama is deployed in ExCL as a module. To use Ollama, load the module, and then you have access to the ollama CLI interface.

Load the Ollama module with:

module load ollama

Ollama has a server component which stores files in its home. This server component should be launched using a service account by ExCL admin, since it provides ollama for the entire system. Ollama is already running on some of the workers in ExCL. See the output from the model load for an up-to-date list. Contact [email protected] if you would like ollama to be available on a specific system.

Ollama API

Examples of using the Ollama API can be found at .

Links

ExCL → User Documentation → Contribute → Authoring Guide

Authoring Guide for ExCL

Perhaps you've got some how-to documents tucked away in folders that you'd like to share with the community. Or maybe you've discovered a way of doing things that would benefit other users.

You can submit your user guides for publication within the ExCL documentation site! See the contributing page for instructions.

We've assembled here the fundamental authoring guidelines for ExCL user documentation.

Document and Content Preferences

• Documents should be created using using the syntax.

• Oak Ridge National Laboratory (ORNL) uses the (CMOS) as a basic style guide.

• Define the first instance of every acronym in each document. Ensure that the long-form is not repeated after it is defined.

• Buttons and links that the user should "click" should go in code

Pictures and Images

Screenshots and images cannot be resized using markdown. Therefore, we embed .html that is rendered when we publish the tutorial to the documentation site.

• Images and screenshots should be stored in a folderassets. Images and screenshots added from the gitbook interface are stored in .gitbook/assets, but issues seem to occur if this folder is modified externally from gitbook.

• Files should be named descriptively. For example, use names such as adding-IP-address.png instead of image03.png.

• To remain consistent with other images in tutorials, please use the following

Other Considerations

• Have you redacted sensitive information from text and images?

• Have you removed information that is protected by copyright?

• Are you using a specific version of your software and have you included in the documentation?

Related Topics

• Using a for creating user content.

The EDA tools are installed on the system dragon. dragon can be access via ssh from the login node, via x11 forwarding from the login node's ThinLink, or directly via ThinLink with Foxy Proxy. See ThinLinc Quickstart to get started with ThinLinc Setup. See Accessing ExCL for more details on logging in.

Ssh access:

ssh -Y -J <username>@login.excl.ornl.gov <username>@dragon

ThinLinc access to login:

https://login.excl.ornl.gov:300

ThinLinc access to dragon (Requires reverse proxy to be setup):

https://dragon.ftpn.ornl.gov:300

All of the tools are installed to /opt/Siemens and the tools can be set up with

Also, please join the siemens-eda slack channel in the ORNL CCSD slack.

Thank you Chen Zhang for the presentation materials to learn about and get started with Marimo. Marimo works well in ExCL and can be set up to work with the Ollama instance running in ExCL to enable the AI features.

Download Marimo Quick-start Presentation

Description

EMU-Chick System is composed of 8x nodes that are connected via RapidIO Interconnect.

Each node has:

• 8x nodelets, array of DRAMs

• A stationary core (SC)

• Migration engine, PCI-Express interfaces, and an SSD.

• 64-byte channel 64GB of DRAM, divided into eight 8-byte narrow-channel-DRAMs (NC-DRAM

Each nodelet has:

• 2x Gossamer cores (GC)

• 64 concurrent in-order, single-issue hardware threads

Access

• The path to access to each individual EMU node is: login.excl.ornl.gov ⇒ emu-gw ⇒ emu ⇒ {n0-n7}

• emu-gw is an x86-based gateway node.

Development Workflow

• The EMU software development kit (SDK) is installed under /usr/local/emu on emu-gw, which is an x86 based system. Compilation and simulation should be performed on this machine.

• The official EMU programming guide is located under /usr/docs.

• emu and emu-gw mount home directories, so you should have no difficulty accessing your projects. Please use $HOME (or ${HOME}) as your home directory in scripts, as the mount location of your home directory, may change.

Other Resources

This document will be updated with additional documentation references and user information as it becomes available.

Contact

Please send assistance requests to [email protected].

Description

This system is generally identical to the nodes (AC922 model 8335_GTW) in the ORNL OLCF Summit system. This system consists of

• 2 POWER9 (2.2 pvr 004e 1202) cpus, each with 22 cores and 4 threads per core.

The MI300A system (host name faraday) is available for ExCL users. As usual you have to log in through the login node.

Make sure that you

to set up all of the environment needed.

A very light test program is available via git at . This is a good way to ensure your environment is set up correctly.

All tests should return err[0]. If they do not, then it is likely that you do not have render group permissions

To check, run the groups command (on faraday) and see if you are in the render group.

If you are not, contact , and we’ll get you in.

pcie

Description

This system is intended for pci-based device support.

This system is a generic development server purchased with the intent of housing various development boards as needed.

I can't log in

There are two most likely sources of this problem

Password won't work

The most frequent cause is having your visitor (non-ORNL internal password) wrong, or having had it expire. See

DevDocs

This guide goes over hosting internal to ORNL documentation using ExCL’s devdocs VM. For an example of a project which uses devdocs, see and ().

The documentation for hunter is built with GitLab-CI. Here are the relevant lines in .gitlab-ci.yml.

Description

The Qualcomm® Snapdragon™ 855 Mobile Hardware Development Kit (HDK) is a highly integrated and optimized Android development platform.

Accessing this system:

See to learn more about Julia.

Use module load julia to load the Julia tooling on an ExCL system.

Julia VSCode Extension in ExCL

Since Julia is install and loaded as a module, the has trouble finding the Julia executable needed to run properly. Therefore to use the extension on ExCL worker nodes via Remote SSH, you must explicitly set the Julia executable location to the correct path.

This can be done by setting the julia.executablePath

These card are currently installed on Secretariat and Affirmed, but will eventually be moved to take advantage of GPUs installed elsewhere.

Software installation

All DOCA, embedded and BSP software was updated in September 2023, using the following:

• doca-host-repo-ubuntu2204_2.2.0-0.0.3.2.2.0080.1.23.07.0.5.0.0_amd64.deb

Hello World built with nvhpc

Load the Nvidia HPC SDK environment module

$ module load nvhpc-openmpi3

Verify the compiler path

$ which mpicc
/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/comm_libs/openmpi/openmpi-3.1.5/bin/mpicc

Build the program

Run the program with MPI

• -np 4 specifies that 4 processes will be created, each running a copy of the mpi_hello_world program

• -mca coll_hcoll_enable 0 disables HCOLL

Notes

InfiniBand and HCOLL

ExCL systems typically do not have InfiniBand setup. (Although if this is required, it can be added as needed.) HCOLL (HPC-X: Collective Communication Library) requires an InfiniBand adapter and since it's enabled by default, you could see HCOLL warnings/errors which state that no HCA device can be found. You can disable HCOLL and get rid of these warnings/errors with the -mca coll_hcoll_enable 0 flag for example: mpirun -np 4 -mca coll_hcoll_enable 0 ./a.out.

Git (code revision management system) is installed on all ExCL systems on which it makes sense. Git operates as expected, except for external access.

If you require access to external git resources, you need to do a little more.

HTTP or HTTPS access

For HTTP or HTTPS access, make sure you have the following environment variables (they should be set by default, but may not be if you have altered your environment)

The proxy server has access to the full Oak Ridge network (open research only).

Git SSH Access

ssh can be used to clone repositories on the login node. In order to clone repositories on the internal nodes, the ssh config needs to be changed to use the login node as a proxy jump. Here is an example ssh config with jump proxies to code.ornl.gov, bitbucket.org, and github.com.

~/.ssh/config:

To configure git to always use ssh for code.ornl.gov repositories, use the config command below.

Setup Git access to

The recommended approach to access code.ornl.gov is to use SSH. To do this, you need to generate an SSH key and add it to your GitLab account. The following steps will guide you through the process.

1. Generate an SSH key.

1. Add the SSH key to your GitLab account.

1. Copy the output of the command and paste it into the SSH key section of your GitLab account settings.

2. If you are on an ExCL system and you have not already done so, configure your SSH client to use the login node as a jump proxy. See for more information.

If you use a passphrase with your SSH key (recommended for security), then you should also setup an SSH Agent to load the SSH key. This allows you to enter your passphrase once for the session without needing to enter your passphrase potentially many times for each git command. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms, see .

SSH Keys for Authentication

Using SSH keys is the preferred way to authenticate your user and to authenticate with private Git repositories. For security, it is recommended to use an SSH keys encrypted with a passphrase.

Why not passwords?

ExCL will block your account after 3 failed attempts. Automatic login tools, e.g. VS Code, can easily exceed this limit using a cached password and auto-reconnect. For git repos with two-factor authentication, an application token/password must be created, and this password must be stored externally and is more cumbersome to use.

How to get started?

1. Set up a key pair:

   • Your ExCL account has an automatically generated SSH key pair created for you on account creation. This key pair allows you to connect to internal nodes from the login node without having to type a password. (If you are having to type a password then this key pair has been messed up.) So one easy option is to copy this private key from ExCL to your local system and then use it to login to ExCL. If you local system does not already have a key pair, then you can copy login.excl.ornl.gov:~/.ssh/id_rsa and login.excl.ornl.gov:~/.ssh/id_rsa.pub to your local ~/.ssh folder. (if you already have a key pair this will override you previous version so make sure to check before copying.) Make sure you chmod 600 these files so that the private key has sufficient permission protection to allow openssh to use the keys. You can also upload your public key to Git websites like code.ornl.gov to push and push git repositories. See .

SSH-Agent and SSH Forwarding

SSH-Agents cache SSH keys with passphrases, allowing them to be reused during the session. This is not needed with keys without a passphrase, since they can be used without decrypting.

SSH Forwarding: SSH agents can forward SSH keys to a remote system, making the keys available there as well.

How to get started?

1. .

2. Add key to agent

   • ssh-add or ssh-add [file] for non-default filenames.

This guide goes over the recommended way to install Conda and Spack in ExCL. If you are already familiar with the Conda and Spack installation process, then these tools can be installed to their default locations. One recommendation is to store the environment.yml and spack.yaml files in your git repositories to make it easy to recreate the Conda and Spack environments required for that project. The remainder of this page goes over the installation in more detail.

Installing Conda

See for the latest installation instructions. Miniforge is the recommended version of conda since its a minimal base install which defaults to using conda-forge packages.

Follow the prompts on the installer screens. Accept the license agreements. (Optional) Specify /home/$USER/conda as the installation location. Choose if you want the installer to initialize Miniforge.

Improving Conda Environment Solver Performance

To improve the performance of the Conda environment solver, you can use the conda-libmamba-solver plugin which allows you to use libmamba, the same libsolv-powered solver used by and , directly in conda.

The quick start guide is below.

See and for more information.

Installing Spack

Groq Links

• Groq API

• GroqFlow

Login and Groq Cards Available

Start by logging into ExCL's login node.

From the login node, you can then login to a node with a Groq card, for example

Here is a table of the Groq cards available:

Groq card and Slurm

The recommended way to access the Groq card is to reserve it through the Slurm resource manager. Groq cards are available on machines in the groq partition. To reserve a node with a groq card for interactive use use the command.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

Non-interactive batch jobs can similarly be launched.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

or specified in the script:

Using the Groq Card

In order to use the Groq API you need to make sure you are using python 3.8 and that you add the Groq python libraries to your path. For python 3.8 you can either use the installed system python3.8 or use conda to install python3.8.

System python3.8

You need to fully quantify the path to python since Ubuntu 22.04 defaults to python3.10. This means you need to use

Then to install jupyter notebook in your home directory, you would need to do

Conda

First install miniconda by following . Then create a groq environment with

See the for more details for setting up the Conda environment.

Graphical Access to Groq Systems

See .

Jupyter Notebooks

See for more information on setting up Jupyter Notebooks within ExCL.

Getting started Resources

Useful Groq Commands

• Run regression tests to verify card functionality: tsp-regression run

• Get Groq device status: /opt/groq/runtime/site-packages/bin/tsp-ctl status

• Monitor temperature and power: /opt/groq/runtime/site-packages/bin/tsp-ctl monitor

oswald00

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald01

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald02

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald03

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

ExCL uses Modules to manage software environments efficiently. Modules allow users to load, unload, and switch between different software versions without modifying system paths manually. Please let us know if there is a software package you would like us to make available via a module.

Loading a Module

To load a specific software module:

Example:

This makes Python 3.9 available for use.

You can also leave off the version number to load the default version.

Example:

Listing Available Modules

To see all available modules:

Checking Loaded Modules

To view currently loaded modules:

Unloading a Module

To remove a specific module:

Example:

Switching Between Versions

To switch from one module version to another:

Example:

Resetting the Environment

To clear all loaded modules and reset to the default environment:

ExCL-Utils

The excl-utils module contains common Rust CLI utilities. Load the module to see an updated list of the installed utilities. Additional Rust CLI utilities can be requested to be included in the module. These utilities are updated to the latest versions nightly.

Register a Runner

Runners can be registered as either a group runner or for a single repository (also know as a project runner). Group runners are made available to all the repositories in a group.

Send the following information to [email protected] and we will register the runner as a system runner.

• URL

• Registration Token

• Executor (choose shell or docker with image)

• Project Name (This can be group name or repo name)

• ExCL System

• Tag List

The method for obtaining this information differs depending on if you want to register a group runner or a single repository runner. See and sections below.

After the runner is added, you can edit the runner to change the tags and description.

Group Runner

Navigate to the group page. Click on Build → Runners. Then select New group runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

Single Repo Runner (Project Runner)

Navigate to the repo page. Click on Settings → CI/CD → Runners. Then select New project runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

List of ExCL Systems with a runner

Any system can be requested as a runner. These systems are already being used as a runner. (Updated October 2023)

• docker.ftpn.ornl.gov

• explorer.ftpn.ornl.gov

• intrepid.ftpn.ornl.gov

• justify.ftpn.ornl.gov

Using Slurm with Gitlab CI

The system slurm-gitlab-runner is setup specifically to run CI jobs that then run the execution using slurm with sbatch --wait.

For a complete example and template for how to use the Slurm with GitLab in ExCL see and .

This template includes two helper scripts, runner_watcher.sh and slurm-tee.py.

runner_watcher.sh watches the CI job and cancels the Slurm job if the CI job is canceled or times out.

slurm-tee.py watches the slurm-out.txt and slurm-err.txt files and prints their content to std-out so that the build log can be watched from the GitLab web interface. Unlike regular less --folow, slurm-tee watches the multiple files for changes and also exits once the slurm job completes.

Spack and Conda with Gitlab-CI

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

Once you have access you have a couple of options.

• login.excl.ornl.gov runs an SSH Server and you can connect to the login node with ssh login.excl.ornl.gov.

• There is a limited number of ThinLinc licenses available. Thinlinc (Xfce Desktop) can be accessed at https://login.excl.ornl.gov:300 for HTML5 services, and ThinLinc clients can use login.excl.ornl.gov as their destination. ThinLinc clients can be downloaded without cost from . ThinLinc provides much better performance than tunneling X over SSH. A common strategy is to access login.excl.ornl.gov via ThinLinc and then use X11 forwarding to access GUIs running on other nodes.

Notes:

• Using an SSH key instead of a password to connect to ExCL is highly recommended. See . SSH keys are more secure than passwords, and you are less likely to accidentally get banned from multiple incorrect login attempts when using SSH Keys to authenticate. If you get blocked, you can send a help ticket to with your IP address to get removed from the block list.

• If you use a passphrase with your SSH key (recommended for security), you should also set up an SSH Agent to load the SSH key. An SSH Agent allows you to enter your passphrase once for the session without needing to enter your passphrase many times. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms; see .

• It is recommended to use a terminal multiplexer like or . These tools keep your session active and can be reattached to if you loose network connection. They also allow you to open multiple windows or split panels.

Add SSH Public Key to ExCL’s Authorized Keys

You can manually copy the key if already on ExCL. For example

Or you can you ssh-copy-id to copy your local systems key to ExCL.

Roadmap for Setup

If you are new to remote development on ExCL here is a roadmap to follow to set important settings and to get familiar with remote Linux development.

1. Access ExCL

2. Setup SSH:

   • Bonus:

3. Setup Git

4. Setup VS Code Remote Explorer:

   • Important: Make sure to check the setting Remote.SSH: Lockfiles in Tmp.

5. . This enables access to as well as any other web services running on ExCL systems.

6. Now you are ready to follow any of the other Quick-Start Guides.

Setup FoxyProxy

1. Launch SOCKS dynamic proxy forwarding to the login node using dynamic forwarding with SSH. On Linux or macOS, via the SSH flag -D

   or in the ssh config add the DynamicForward option

   On Windows, use MobaSSHTunnel to set up Dynamic Forwarding. See for more information on port forwarding in windows.

2. Setup FoxyProxy Install the FoxyProxy or .

   Setup FoxyProxy by adding a new proxy for localhost on port 9090. Then add the regular expression URL pattern .*\.ftpn\.ornl\.gov

If you don’t want to run the runner as service then you can follow the steps posted at to create a self-hosted runner in ExCL.

Setup Runner as a service in ExCL

If you do want to register the runner as a service, the easiest way is to use systemd user services. To set this up follow the steps below.

This page covers a few recommendations and tips for getting started with Python in ExCL following best practices for packaging python projects and using virtual environments. There are many different ways to structure and package python projects and various tools that work with python, so this page is not meant to be comprehensive but to provide a few recommendations for getting started.

Python Virtual Environments with venv

Using virtual environments is the recommended way to isolate Python dependencies and ensure compatibility across different projects. Virtual environments prevent conflicts between packages required by different projects and simplify dependency management. The goal with isolated, project specific python environments is to avoid the situation found in .

→ → →

Git Scenarios

This document includes common Git scenarios and how to deal with them.

→ → →

Git Workflow from the Command Line

There are many reasons one would prefer to work from the command line. Regardless of your reasons, here is how to contribute to the ExCL documentation using only command line tools.

Jump to a Section:

ExCl → User Documentation → Jupyter Quick Start

Installing Jupyter

Since there are many ways to install Jupyter using various python management tools, I will not reproduce the documentation here. The official documentation for installing Jupyter can be found at Project Jupyter | Installing Jupyter. However, I will highlight the methods of using Jupyter with UV, running Jupyter Notebooks in VS Code, and the alternative to Jupyter notebooks, Marimo | ExCL User Docs. These methods are all the methods that I typically use when working with python notebooks.

Jupyter with UV

See the UV documentation, . This documentation is well written and covers:

Jupyter kernels using virtual environments

See . Although , the following steps are still a good way to manually create and use a kernel from Jupyter.

Create a python virtual environment and activate it. Then install ipykernel and then install the kernel for use in Jupyter.

Use jupyter kernelspec list to view all the installed Jupyter kernels.

To uninstall a Jupyter kernel use uninstall.

Accessing a Jupyter Notebook Running on ExCL

A Jupyter notebook server running on ExCL can be accessed via a local web browser through port forwarding the Jupyter notebook's port. By default, this is port 8888 (or the next available port). This port might be in use if someone else is using running a notebook. You can specify the port with the --port flag when launching the Jupyter notebook. To use a different port just replace 8888 with the desired port number. In order to port forward from an internal node, you have to port forward twice, once from your machine to login.excl.ornl.gov and once again from the login node to the internal node (i.e. pcie).

Detailed instructions for Linux/Mac

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster. If you want to access a different system, then replace pcie with the system you intend to access.

1. Specify the ports that you intend to use. Choose a different number from the default so that you don't conflict with other users.

2. From your local machine connect to pcie using login.excl.ornl.gov as a proxy and local forward the jupyter port.

3. (Optional) Load the anaconda module if you don't have jupyter notebook installed locally.

4. Launch the Jupyter server on pcie

If you ssh client is too old for proxyjump to work, you can always break up the process into another step.

1. From your local machine connect to login.excl.ornl.gov and local port forward port 8888.

2. From the login node connect to pcie and local port forward port 8888

3. Launch the Jupyter server on pcie

4. Connect to the Jupyter notebook using a web browser on your local machine. Use the token shown in the output from running the Jupyter server. Url: http://localhost:8888/?token=<token>

Detailed instructions for Windows with MobaXterm

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster.

1. From your local machine connect to login.excl.ornl.gov using MobaXterm.
2. Go to tools and click on MobaSSHTunnel. Use MobaSSHTunnel local forward port 8888.

   Click on MobaSSHTunnel

   Click on New SSH Tunnel

Detailed instructions for Windows with Visual Studio Code

These instructions go over how to access a Jupyter notebook running on the quad00 node in the ExCL Cluster using Visual Studio Code to handle port forwarding.

1. Open Visual Studio Code

2. Make sure you have the Remote - SSH extension installed.
3. Setup .ssh

   Navigate to the remote explorer settings.

These 2U servers are highly capable large memory servers, except that they have limited PCIe4 slots for expansion.

• HPE ProLiant DL385 Gen10 Plus chassis

• 2 AMD EPYC 7742 64-Core Processors

  • configured with two threads per core, so presents as 256 cores

  • this can be altered per request

• 1 TB physical memory

  • 16 DDR4 Synchronous Registered (Buffered) 3200 MHz 64 GiB DIMMS

• 2 HP EG001200JWJNQ 1.2 TB SAS 10500 RPM Disks

  • one is system disk, one available for research use

• 4 MO003200KWZQQ 3.2 TB NVME storage

  • available as needed

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM, and for networking/DPU research.

Status

Affirmed

Affirmed is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Justify

Justify is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Pharaoh

Pharaoh is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Secretariat

Secretariat is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

is the most widely used container system for HPC. It is designed to execute applications at bare-metal performance while being secure, portable, and 100% reproducible. Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work.

is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Apptainer allows for more secure containers than docker without the need for root access.

is a lightweight but powerful source code editor which runs on your desktop and is available for Windows, macOS, and Linux. The editor has IntelliSense, debugger support, built-in git, and many extensions to add additional support to the editor. VSCode supports and development on remote servers via ssh. Plugins add language support, linters, and compilers for many languages including Python, C/C++, CMake, and markdown.

Remote Explorer

The login node has install and can be accessed at . Since this node is public facing, it is the easiest to access with ThinLinc.

In addition to the login node, multiple systems including the have installed, which makes it easier to run graphical applications. To access ThinLinc you need to use as socks proxy to forward traffic to the ExCL network or port forwarding of port 22 to use the ThinLinc client.

For better keyboard shortcut support and to prevent the browser from triggering the shortcuts, I recommend installing .

Zenith 1

Created using PC Part Picker. The build is available at https://pcpartpicker.com/list/xPkRwc.

PCPartPicker Part List

Access to the GPUs

To have access to the GPUs, request to be added to the video and render groups if you are not already in these groups.

Images

Zenith 2

Created using PC Part Picker. The build is available at .

Images

Arty A7

To get started, first install the . I recommend using the git approach. Once setup you can load the Vitis module and start developing with Vivado. See . To reserve the hardware use Slurm like normal, see .

ExCl → User Documentation → Contributing → Git Basics

Git Basics

Git, like other version control (VC) software/system (see a Wikipedia list), tracks changes to a file system over time. It is typically used in software development but can be used to monitor changes in any file.

Git - a version control system that records the changes to a file or files which allows you to return to a previous version

📝 Note: This tutorial uses only the command line. After you have learned the basics of Git, you can explore a Git workflow , or , and also, common .

ORNL Git Resources

When we talk about Git, we say that a repository stores files. This term means that you have a folder that is currently being tracked by Git. It is common, although optional, to use one of the Git repository (repo) services (GitHub, GitLab, BitBucket, etc.). You could easily set up Git tracking on your local machine only, but one of the perks to using Git is that you can share your files with others and a team can edit files collaboratively. The ability to collaborate is one of the many reasons why hosted Git repos are so popular.

Repository - the Git data structure which contains files and folders, as well as how the files/folders have changed over time

ORNL provides two GitLab servers and , the latter being accessible only inside of ORNL. Project owners control access to GitLab repositories. You may log in and create your projects and repositories, and share them with others.

Accessing GitLab

• In your browser, navigate to and login using your UCAMS credentials. Click on the green button at the top of the window that says New project.

• Choose the Blank project tab, create a name for the project, and select the "Visibility Level" that you prefer. Then click Create project.

• Notice that GitLab has provided instructions to perform Git setup and initialization of your repository. We will follow those instructions.

Local Machine Setup

• First, use the command line to see if Git is installed. (Windows users may check their list of currently installed programs.)

  • To install or update Git using your package manager:

    • CentOS, RedHat:

Using Branches to Make Changes

Branches are created as a way to separate content that is still under development. One way to think about a branch is as a copy of the content of a repository at a point in time. You'll then make your changes on the copy before then integrating the changes back into the original. For example, if you were using your GitLab repo to host a website, you probably would not want incomplete content shown to those who would visit your site. Instead, you can create a branch, make edits to the files there, then merge your development branch back into the master branch, which is the default branch. Additionally, branches are commonly used when multiple individuals work out of a single repository.

Branch - a version of the repository that splits from the primary version Merge - using the changes from one branch and adding them to another

• A branch checkout enables you to make changes to files without changing the content of the master branch. To create and checkout a branch called "adding-readme":

Checkout - Git command to change branches

• Now we edit the README.md file to add a description of the repository. The file needs to be opened with a text editor (nano, vim, emacs, etc.).

• Add your description. README.md is a markdown file. If you do not know how to use markdown, don't worry. Basic text works, too. However, if you would like to learn markdown, it is simple. .

  • To type in

Merging Content from a Development Branch to the Master Branch

After completing the previous section, we have two branches: adding-readme and master. We are ready to move the adding-readme content to the master branch.

You can create a merge request using the GitLab GUI.

• From the left menu panel in Gitlab (when viewing the repository), select Merge Request then the green New merge request button.

• Select your branch on the "Source Branch" side (adding-readme).

  • Target branch is

From the left menu panel in Gitlab, select Overview to see the new README.md content.

External Reference Material

Sometimes Git repository sites use different terminology, i.e., merge request vs. pull request. To reference the glossaries:

Ready to Learn More?

Compilers are, in general, maintained from a central NFS repository, and made accessible via the module command (from Lmod). For example

hsm@secretariat:~$ module load gnu
hsm@secretariat:~$ module avail

---------------- /usr/share/lmod/lmod/modulefiles ----------------
   Core/lmod/6.6    Core/settarg/6.6

------ /auto/software/swtree/ubuntu20.04/x86_64/modulefiles ------
   anaconda/3          git/2.38.0             julia/1.8.0
   cmake/3.22.5        gnu/10.2.0             llvm/8.0.1
   gcc/10.2.0          gnu/11.1.0             llvm/13.0.1
   gcc/11.1.0          gnu/11.3.0             llvm/14.0.0 (D)
   gcc/11.3.0          gnu/12.1.0    (L,D)
   gcc/12.1.0   (D)    hipsycl/0.9.2

  Where:
   L:  Module is loaded
   D:  Default Module

If you do not load a module, you will get the default compiler as delivered by the operating system vendor (4.8.5 on some systems). If you module load gnu you will currently get 12.1.0, as it is the default. If you need, say, 10.2.0, you need to module load gnu/10.2.0. Note that documentation details with respect to compiler availability and versions will not necessarily be kept up to date; the system itself is authoritative.

Some compilers (notably xlc and the nvhpc tool chain) cannot be installed on nfs, so if they are available they will show up in a different module directory. The same module commands are used.

Additional compilers can be installed on request to [email protected]. Maintaining multiple Gnu suites is straightforward, less so for other tool suites.

Additional compilers and tools can also be installed using Spack.