This is the user documentation repository for the Experimental Computing Laboratory (ExCL) at Oak Ridge National Laboratory.

This site is undergoing development; systems and processes will be documented here as the documentation is created.

See the index on the left of this page for further detail.

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated. See Acknowledgment for details.

See Requesting access for information on how to request access to the system.

How to Login

See for more details.

• Shell login: ssh login.excl.ornl.gov

• ThinLinc Session:

Getting Assistance

Please send an email request to for assistance. This initiates a service ticket and dispatches it to ExCL staff.

ExCL Cheat Sheet

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated.

Sample acknowledgment:

This research used resources of the Experimental Computing Laboratory (ExCL) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725

You may use any variation on this theme, calling out specific simulations or portions of the research that used ExCL resources, or citing specific resources used.

However, the crucial elements to include are:

The Experimental Computing Laboratory is a Advanced Computing Systems Research project directed by Jeffrey Vetter. Support staff include:

• Steve Moulton - systems engineer

• Aaron Young - software engineer

Contact [email protected] for assistance.

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

ExCL reserves the first Tuesday of every month for systems maintenance. This may result in complete inaccessibility during business hours. Every effort will be made to minimize the scope, duration, and effect of maintenance activities.

If an outage will affect urgent projects (i.e., with impending deadlines) please email [email protected] as soon as possible.

Two Nvidia H100s are now available on hudson.ftpn.ornl.gov. From Nvidia documentation:

The NVIDIA H100 NVL card is a dual-slot 10.5 inch PCI Express Gen5 card based on the NVIDIA Hopper™ architecture. It uses a passive heat sink for cooling, which requires system airflow to operate the card properly within its thermal limits. The NVIDIA H100 NVL operates unconstrained up to its maximum thermal design power (TDP) level of 400 W to accelerate applications that require the fastest computational speed and highest data throughput. The NVIDIA H100 NVL debuts the world’s highest PCIe card memory bandwidth of nearly 4,000 gigabytes per second (GBps)

Basic validation has been done via running the nvidia samples nbody program on both devices:

10485760 bodies, total time for 10 iterations: 401572.656 ms
= 2738.014 billion interactions per second
= 54760.284 single-precision GFLOP/s at 20 flops per interaction

The GPUs are available to the same UIDs as are using the A100s on milan0. If nvidia-smi does not work for you, you don't have the proper group memberships -- please send email to [email protected] and we will fix it. nvhpc is installed as a module as it is on other systems.

Currently has a U250 installed with a custom application deployed which requires an older linux kernel.

Lewis is configured with kernel 5.15.0.

Hold set with:

apt-mark hold 5.15.0-72-generic

To remove hold:

apt-mark unhold 5.15.0-72-generic

Thank you Chen Zhang for the presentation materials to learn about and get started with Marimo. Marimo works well in ExCL and can be set up to work with the Ollama instance running in ExCL to enable the AI features.

Download Marimo Quick-start Presentation

→ →

About ExCl User Documentation

Documentation published to ExCL users is available in our . Users are encouraged to contribute by improving the material or providing user-created tutorials to share with the community.

Roadmap for Setup

If you are new to remote development on ExCL here is a roadmap to follow to set important settings and to get familiar with remote Linux development.

1. Setup SSH:

The EDA tools are installed on the system dragon. dragon can be access via ssh from the login node, via x11 forwarding from the login node's ThinLink, or directly via ThinLink with Foxy Proxy. See to get started with ThinLinc Setup. See for more details on logging in.

Ssh access:

ThinLinc access to login:

See documentation at https://code.ornl.gov/excl-devops/documentation/-/tree/master/devops.

See also the Gitlab-CI in ExCL Quick-start Guide.

Description

EMU-Chick System is composed of 8x nodes that are connected via RapidIO Interconnect.

Each node has:

• 8x nodelets, array of DRAMs

• A stationary core (SC)

• Migration engine, PCI-Express interfaces, and an SSD.

• 64-byte channel 64GB of DRAM, divided into eight 8-byte narrow-channel-DRAMs (NC-DRAM

Each nodelet has:

• 2x Gosamer cores (GC)

• 64 concurrent in-order, single-issue hardware threads

Access

• The path to access to each individual EMU node is: login.excl.ornl.gov ⇒ emu-gw ⇒ emu ⇒ {n0-n7}

• emu-gw is an x86-based gateway node.

Development Workflow

• The EMU software development kit (SDK) is installed under /usr/local/emu on emu-gw, which is an x86 based system. Compilation and simulation should be performed on this machine.

• The official EMU programming guide is located under /usr/docs.

• emu and emu-gw mount home directories, so you should have no difficulty accessing your projects. Please use $HOME (or ${HOME}

Other Resources

This document will be updated with additional documentation references and user information as it becomes available.

Contact

Please send assistance requests to [email protected].

Description

This system is generally identical to the nodes (AC922 model 8335_GTW) in the ORNL OLCF Summit system. This system consists of

• 2 POWER9 (2.2 pvr 004e 1202) cpus, each with 22 cores and 4 threads per core.

• 6 Tesla V100-SXM2-16GB GPUs

• 606GiB memory

• automounted home directory (on group NFS server)

Contact

• [email protected]

Usage

As currently configured this system is usable using conventional ssh logins (from login.excl.ornl.gov), with automounted home directories. GPU access is currently cooperative; a scheduling mechanism and scheduled access is in design.

The software is as delivered by the vendor, and may not be satisfactory in all respects as of this writing. The intent is to provision a system that is as similar in all respects to Summit, but some progress is required to get there. This is to be considered an early access machine.

Please send assistance requests to [email protected].

Installed Compilers

Please see

GPU Performance

This system is still being refined with respect to cooling. As of today, rather than running at the fully capable 300 watts per GPU, GPU usage has been limited to 250 watts to prevent overheating. As cooling is improved, this will be changed back to 300 watts with dynamic power reduction (with notification) as required to protect the equipment.

It is worth noting that this system had to be pushed quite hard (six independent nbody problems, plus CPU stressors on all but 8 threads) to trigger high temperature conditions. These limits may not be encountered in actual use.

Performance Information

GPU performance information can be viewed at

Request access by emailing [email protected].

Other Resources

• IBM 8335-GTW documentation:

pcie

Description

This system is intended for pci-based device support.

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Atipa

• Tyan Motherboard S7119GMR-06

• 192 GB memory

• Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHzIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz 2x16 cores no hyperthreading

Use

This system is used for heterogeneous accelerator exploration and FPGA Alveo/Vitis-based development.

Current VMs

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

Description

The Qualcomm® Snapdragon™ 855 Mobile Hardware Development Kit (HDK) is a highly integrated and optimized Android development platform.

Accessing this system:

• Qualcomm board is connected to an HPZ820 workstation (McMurdo) or to an HP Z4 workstation (Clark) through USB

• Development Environment: Android SDK/NDK

• Login to mcmurdo or clark

  • $ ssh –Y mcmurdo

• Setup Android platform tools and development environment

  • $ source /home/nqx/setup_android.source

• Make sure you have a functining environment

  • adb kill-server

  • adb start-server

• Run Hello-world on ARM cores

  • $ git clone

  • $ make compile push run

• Run OpenCL example on GPU

  • $ git clone

  • Run Sobel edge detection

Other Details

The snapdragon SDK uses python 2.7; you may need to explicitly specify python2 in your environment.

Access

Access will be granted per request (as this cannot be used as a shared resource).

Useful Links

1. Android Studio:

2. Qualcomm HDK:

3. Quallcomm Neural Processor SDK:

Images

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

Once you have access you have a couple of options.

• login.excl.ornl.gov runs an SSH Server and you can connect to the login node with ssh login.excl.ornl.gov.

• There is a limited number of licenses available. Thinlinc (Xfce Desktop) can be accessed at for HTML5 services, and ThinLinc clients can use login.excl.ornl.gov as their destination. ThinLinc clients can be downloaded without cost from . ThinLinc provides much better performance than tunneling X over SSH. A common strategy is to access login.excl.ornl.gov via ThinLinc and then use X11 forwarding to access GUIs running on other nodes.

Notes:

• Using an SSH key instead of a password to connect to ExCL is highly recommended. See . SSH keys are more secure than passwords, and you are less likely to accidentally get banned from multiple incorrect login attempts when using SSH Keys to authenticate. If you get blocked, you can send a help ticket to with your IP address to get removed from the block list.

• If you use a passphrase with your SSH key (recommended for security), you should also set up an SSH Agent to load the SSH key. An SSH Agent allows you to enter your passphrase once for the session without needing to enter your passphrase many times. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms; see .

• It is recommended to use a terminal multiplexer like

Add SSH Public Key to ExCL’s Authorized Keys

You can manually copy the key if already on ExCL. For example

Or you can you ssh-copy-id to copy your local systems key to ExCL.

See the Cheat Sheet for a quick summary.

Backup

User files (home directories) are stored on an ZFS-based NFS server, and are generally available to all ExCL systems (there are exceptions for operational and security reasons; if you trip over something please let [email protected] know). The /noback/<user> facility is no longer supported and is not being created for new users accounts. Files already in the /noback hierarchy will not be affected; if you would like assistance in moving these files to your home directory please let [email protected] know. Space available to /noback is limited.

ExCL uses ZFS with snapshots. Zrepl () handles both automated snapshot generation and file system replication. Snapshots are taken hourly, and ExCL file systems are replicated to the back up (old FS00) fileserver. The snapshot directory name format is ~/.zfs/snapshots/zrepl_yyyymmdd_hhmmss_000 (where the hour is in UCT not Eastern Daylight/Standard Time). The use of UCT in the snapshot name is a zrepl property to enable global replication consistency, and is not modifiable. If you deleted or made a destructive modification to, say, ~/.bashrc on, say, June 11, 2024 at 3 PM, it should be available in ~/.zfs/snapshots/zrepl_20240611_185313_000/.bashrc, and in earlier snapshots.

Snapshots take space for files that have changed or been deleted. They are automatically deleted as they age, so that hourlies are kept for 48 hours, one hourly from each day is kept for 30 days, and one hourly for each 30 day period is kept for 180 days. This policy can be modified on request. Snapshots are read only; you can copy files from them back into your home directory tree to restore them.

There is currently no file purge policy. Given that ExCL researchers take care of cleaning up files that are no longer in use, no change to this policy is foreseen. Files for inactive users are archived in a non-snapshot file system. While it is our intent to continue maintaining storage for inactive users, this policy may change in the future.

Quotas

Refquotas are applied to the ZFS filesystems to avoid runaway storage usage. A refquota limit applies only to your files, not to snapshot storage. ZFS stores data in a (very fast) compressed format, so disk usage may appear to be less than you expect. Home and project subvolumes start with a refquota of 512G. Users can request higher quotas via . We can also help diagnose the cause of large storage use by providing a breakdown of file usage and helping clean up unneeded large files and snapshots.

Local Storage

In addition to shared network storage, each system has a local /scratch directory. The size will vary from system to system, and some systems may have /scratch2 in addition. A working space can be created with mkdir /scratch/$USER if one is not already present. This storage location is good for caching files on local host storage, for speeding up tasks which are storage IO bound, and performing tasks which fail on NFS storage (for example, Apptainer and embedded Linux builds). If you require more scratch storage than is available contact as on newer systems there is often additional storage available that has not been allocated. Similarly contact us if there is no /scratch or /scratch2 directory. Since there is (currently) no purging policy, please clean up after you no longer need your scratch space.

DevDocs

This guide goes over hosting internal to ORNL documentation using ExCL’s devdocs VM. For an example of a project which uses devdocs, see and ().

The documentation for hunter is built with GitLab-CI. Here are the relevant lines in .gitlab-ci.yml.

Hello World built with nvhpc

Load the Nvidia HPC SDK environment module

$ module load nvhpc-openmpi3

Verify the compiler path

Build the program

Run the program with MPI

• -np 4 specifies that 4 processes will be created, each running a copy of the mpi_hello_world program

• -mca coll_hcoll_enable 0 disables HCOLL

Notes

InfiniBand and HCOLL

ExCL systems typically do not have InfiniBand setup. (Although if this is required, it can be added as needed.) HCOLL (HPC-X: Collective Communication Library) requires an InfiniBand adapter and since it's enabled by default, you could see HCOLL warnings/errors which state that no HCA device can be found. You can disable HCOLL and get rid of these warnings/errors with the -mca coll_hcoll_enable 0 flag for example: mpirun -np 4 -mca coll_hcoll_enable 0 ./a.out.

Register a Runner

Runners can be registered as either a group runner or for a single repository (also know as a project runner). Group runners are are made available to all the repositories in a group.

Send the following information to [email protected] and we will register the runner as a system runner.

• URL

• Registration Token

• Executor (choose shell or docker with image)

• Project Name (This can be group name or repo name)

• ExCL System

• Tag List

The method for obtaining this information differs depending on if you want to register a group runner or a single repository runner. See and sections below.

After the runner is added, you can edit the runner to change the tags and description.

Group Runner

Navigate to the group page. Click on Build → Runners. Then select New group runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

Single Repo Runner (Project Runner)

Navigate to the repo page. Click on Settings → CI/CD → Runners. Then select New project runner and progress until you have created the runner and are provided with a command to run in the command line to register the runner. Since we use system runners instead of user runners, you will need to send this information to to get the runner registered.

List of ExCL Systems with a runner

Any system can be requested as a runner. These systems are already being used as a runner. (Updated October 2023)

• docker.ftpn.ornl.gov

• explorer.ftpn.ornl.gov

• intrepid.ftpn.ornl.gov

Using Slurm with Gitlab CI

The system slurm-gitlab-runner is setup specifically to run CI jobs that then run the execution using slurm with sbatch --wait.

For a complete example and template for how to use the Slurm with GitLab in ExCL see and .

This template includes two helper scripts, runner_watcher.sh and slurm-tee.py.

runner_watcher.sh watches the CI job and cancels the Slurm job if the CI job is canceled or times out.

slurm-tee.py watches the slurm-out.txt and slurm-err.txt files and prints their content to std-out so that the build log can be watched from the GitLab web interface. Unlike regular less --folow, slurm-tee watches the multiple files for changes and also exits once the slurm job completes.

Spack and Conda with Gitlab-CI

Compilers are, in general, maintained from a central NFS repository, and made accessible via the module command (from Lmod). For example

hsm@secretariat:~$ module load gnu
hsm@secretariat:~$ module avail

---------------- /usr/share/lmod/lmod/modulefiles ----------------
   Core/lmod/6.6    Core/settarg/6.6

------ /auto/software/swtree/ubuntu20.04/x86_64/modulefiles ------
   anaconda/3          git/2.38.0             julia/1.8.0
   cmake/3.22.5        gnu/10.2.0             llvm/8.0.1
   gcc/10.2.0          gnu/11.1.0             llvm/13.0.1
   gcc/11.1.0          gnu/11.3.0             llvm/14.0.0 (D)
   gcc/11.3.0          gnu/12.1.0    (L,D)
   gcc/12.1.0   (D)    hipsycl/0.9.2

  Where:
   L:  Module is loaded
   D:  Default Module

If you do not load a module, you will get the default compiler as delivered by the operating system vendor (4.8.5 on some systems). If you module load gnu you will currently get 12.1.0, as it is the default. If you need, say, 10.2.0, you need to module load gnu/10.2.0. Note that documentation details with respect to compiler availability and versions will not necessarily be kept up to date; the system itself is authoritative.

Some compilers (notably xlc and the nvhpc tool chain) cannot be installed on nfs, so if they are availble they will show up in a different module directory. The same module commands are used.

Additional compilers can be installed on request to [email protected]. Maintaining multiple Gnu suites is straightforward, less so for other tool suites.

Additional compilers and tools can also be installed using .

oswald00

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald01

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald02

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

oswald03

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to [email protected].

The login node has ThinLinc install and can be accessed at https://login.excl.ornl.gov:300. Since this node is public facing, it is the easiest to access with ThinLinc.

In addition to the login node, multiple systems including the virtual systems have ThinLinc installed, which makes it easier to run graphical applications. To access ThinLinc you need to use as socks proxy to forward traffic to the ExCL network or port forwarding of port 22 to use the ThinLinc client.

For better keyboard shortcut support and to prevent the browser from triggering the shortcuts, I recommend installing Open-as-Popup.

Systems Availible

Accessing ThinLinc through the web interface

1. and make sure to have the SOCKS dynamic proxy running.

2. Connect to the ThinLinc server using the links above.

Accessing ThinLinc through ThinLinc Client

This approach is recommended if you need better keyboard forwarding support for keyboard shortcuts that are not working with the Web client. The web client approach is easier to use and enables connecting to multiple systems at a time.

If the system is directly accessible (for example login.excl.ornl.gov), then you can specify the system and connect directly.

If the system is an internal node, then local port forwarding must be used. The steps to setting this up are as follows.

1. Forward port 22 from the remote system to your local system through login. On Linux or macOS

   On windows use ssh via powershell, MobaSSHTunnel, Visual Studio Code, or putty to forward port 22. See for more information on port forwarding in windows.

2. Add alias in hosts file for the remote node. This is needed because of how ThinLinc establishes the remote connected. On Linux this host file is /etc/hosts. On windows the file is C:\Windows\System32\drivers\etc\hosts. Host file:

Potential Issues you may encounter

If you use Gnome and do not have access to the module command when you start a terminal session over ThinLinc web, then your terminal session may not be configured as a login session. To resolve

1. Right click on the terminal icon on the left side of your screen

2. In Preferences -> Unnamed, make sure Run command as a login shell is checked.

You will then get login processing (including sourcing the /etc/profiles.d files) and so the module command will now be present.

This guide goes over the recommended way to install Conda and Spack in ExCL. If you are already familiar with the Conda and Spack installation process, then these tools can be installed to their default locations. One recommendation is to store the environment.yml and spack.yaml files in your git repositories to make it easy to recreate the Conda and Spack environments required for that project. The remainder of this page goes over the installation in more detail.

Installing Conda

See the for the latest installation instructions. I install Miniconda instead of Anaconda since I do not require the 3GB of included packages that come with Anaconda and I will be installing my own packages anyways.

Follow the prompts on the installer screens. Accept the license agreements. Specify /home/$USER/conda as the installation location. Choose if you want the installer to initialize Miniconda.

Improving Conda Environment Solver Performance

To improve the performance of the Conda environment solver, you can use the conda-libmamba-solver plugin which allows you to use libmamba, the same libsolv-powered solver used by and , directly in conda.

The quick start guide is below.

See and for more information.

Installing Spack

ExCl → User Documentation → Contribute → Authoring Guide

Authoring Guide for ExCl

Perhaps you've got some how-to documents tucked away in folders that you'd like to share with the community. Or maybe you've discovered a way of doing things that would benefit other users.

You can submit your user guides for publication within the ExCl documentation site! See the page for instructions.

We've assembled here the fundamental authoring guidelines for ExCl user documentation.

Document and Content Preferences

• Documents should be created using using the syntax.

• Oak Ridge National Laboratory (ORNL) uses the (CMOS) as a basic style guide.

• Define the first instance of every acronym in each document. Ensure that the long-form is not repeated after it is defined.

Pictures and Images

Screenshots and images cannot be resized using markdown. Therefore, we embed .html that is rendered when we publish the tutorial to the documentation site.

• Images and screenshots should be stored in a folderassets. Images and screenshots added from the gitbook interface are stored in .gitbook/assets, but issues seem to occure if this folder is modified externally from gitbook.

• Files should be named descriptively. For example, use names such as adding-IP-address.png instead of image03.png.

Other Considerations

• Have you redacted sensitive information from text and images?

• Have you removed information that is protected by copyright?

• Are you using a specific version of your software and have you included in the documentation?

Related Topics

• Using a for creating user content.

is deployed in ExCL as a module. To use Ollama, load the module, and then you have access to the ollama CLI interface.

Load the Ollama module with:

Ollama has a server component which stores files in its home. This server component should be launched using a service account by ExCL admin, since it provides ollama for the entire system. Ollama is already running on some of the workers in ExCL. See the output from the model load for an up-to-date list. Contact if you would like ollama to be available on a specific system.

Git (code revision management system) is installed on all ExCL systems on which it makes sense. Git operates as expected, except for external access.

If you require access to external git resources, you need to do a little more.

HTTP or HTTPS access

For HTTP or HTTPS access, make sure you have the following environment variables (they should be set by default, but may not be if you have altered your environment)

The proxy server has access to the full Oak Ridge network (open research only).

Git SSH Access

ssh can be used to clone repositories on the login node. In order to clone repositories on the internal nodes, the ssh config needs to be changed to use the login node as a proxy jump. Here is an example ssh config with jump proxies to code.ornl.gov, bitbucket.org, and github.com.

To configure git to always use ssh for code.ornl.gov repositories, use the config command below.

Setup Git access to

The recommended approach to access code.ornl.gov is to use SSH. To do this, you need to generate an SSH key and add it to your GitLab account. The following steps will guide you through the process.

1. Generate an SSH key.

1. Add the SSH key to your GitLab account.

1. Copy the output of the command and paste it into the SSH key section of your GitLab account settings.

2. If you are on an ExCL system and you have not already done so, configure your SSH client to use the login node as a jump proxy. See for more information.

If you use a passphrase with your SSH key (recommended for security), then you should also setup an SSH Agent to load the SSH key. This allows you to enter your passphrase once for the session without needing to enter your passphrase potentially many times for each git command. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms, see .

SSH Keys for Authentication

Using SSH keys is the preferred way to authenticate your user and to authenticate with private Git repositories. For security, it is recommended to use an SSH keys encrypted with a passphrase.

Why not passwords?

ExCL will block your account after 3 failed attempts. Automatic login tools, e.g. VS Code, can easily exceed this limit using a cached password and auto-reconnect. For git repos with two-factor authentication, an application token/password must be created, and this password must be stored externally and is more cumbersome to use.

How to get started?

1. Set up a key pair:

   • Your ExCL account has an automatically generated SSH key pair created for you on account creation. This key pair allows you to connect to internal nodes from the login node without having to type a password. (If you are having to type a password then this key pair has been messed up.) So one easy option is to copy this private key from ExCL to your local system and then use it to login to ExCL. If you local system does not already have a key pair, then you can copy login.excl.ornl.gov:~/.ssh/id_rsa and login.excl.ornl.gov:~/.ssh/id_rsa.pub to your local ~/.ssh folder. (if you already have a key pair this will override you previous version so make sure to check before copying.) Make sure you chmod 600 these files so that the private key has sufficient permission protection to allow openssh to use the keys. You can also upload your public key to Git websites like code.ornl.gov to push and push git repositories. See

SSH-Agent and SSH Forwarding

SSH-Agents cache SSH keys with passphrases, allowing them to be reused during the session. This is not needed with keys without a passphrase, since they can be used without decrypting.

SSH Forwarding: SSH agents can forward SSH keys to a remote system, making the keys available there as well.

How to get started?

1. .

2. Add key to agent

   • ssh-add or ssh-add [file] for non-default filenames.

I can't log in

There are two most likely sources of this problem

Password won't work

The most frequent cause is having your visitor (non-ORNL internal password) wrong, or having had it expire. See https://xcams.ornl.gov to address this. If you are ORNL staff, a frequent cause is a failure to keep your internal ORNL systems password up to date (UCAMS) or have missed required training. ExCL makes the same check that any ORNL system makes as to whether a password is valid or an account exists (you will not be able to differentiate the two errors based on the login failure). This will look like

Too many login attempts from a given IP address

ExCL limits logins to five consecutive failures within a short period of time. After that limit exceeded, login atempts from your IP address will be blocked. This might look like

To have this addressed, report your IP address to [email protected]. If you are on an ORNL network, you can use the usual native tools on your system to find your IP address. If you are at home and on a network using NAT (as most home networks do) use to determine your public IPv4 address when external to the lab. Note that this will not report the correct address if you are on an ORNL (workstations or visitor) network.

I can’t clone my git repo

The recommended approach for accessing git repositories in ExCL is to use the SSH protocol instead of the HTTPS protocol for private repositories and either protocol for public repositories. However, both approaches will work with the proper proxies, keys, applications passwords, and password managers in place.

To use the SSH protocol you must first setup SSH keys to the git website (i.e. GitLab, GitHub, and Bitbucket). See for details for how to do this for code.ornl.gov. The other Git Clouds have similar methods to add SSH keys to your profile.

Since the worker nodes are behind a proxy. You must setup an SSH jump host in your .ssh/config to access Git SSH servers. See to verify that you have setup the proper lines in your SSH Config.

I need a newer version of pip

See for instructions on how to setup a Python virtual environment with the latest version of pip.

I need a newer version of Python

See for instructions on how to use UV to setup a Python virtual environment with a specific python version.

is the most widely used container system for HPC. It is designed to execute applications at bare-metal performance while being secure, portable, and 100% reproducible. Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work.

is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Apptainer allows for more secure containers than docker without the need for root access.

If you don’t want to run the runner as service then you can follow the steps posted at to create a self-hosted runner in ExCL.

Setup Runner as a service in ExCL

If you do want to register the runner as a service, the easiest way is to use systemd user services. To set this up follow the steps below.

See to learn more about Julia.

Use module load julia to load the Julia tooling on an ExCL system.

Julia VSCode Extension in ExCL

Since Julia is install and loaded as a module, the has trouble finding the Julia executable needed to run properly. Therefore to use the extension on ExCL worker nodes via

These card are currently installed on Secretariat and Affirmed, but will eventually be moved to take advange of GPUs installed elsewhere.

Software installation

All DOCA, embedded and BSP software was updated in September 2023, using the following:

This page covers a few recommendations and tips for getting started with Python in ExCL following best practices for packaging python projects and using virtual environments. There are many different ways to structure and package python projects and various tools that work with python, so this page is not meant to be comprehensive but to provide a few recommendations for getting started.

Python Virtual Environments with venv

Using virtual environments is the recommended way to isolate Python dependencies and ensure compatibility across different projects. Virtual environments prevent conflicts between packages required by different projects and simplify dependency management. The goal with isolated, project specific python environments is to avoid the situation found in https://xkcd.com/1987/.

If you are a using the , the simple function show below is a wrapper around venv to activate a python virtual environment if one already exists in .venv in the current directory or create a new virtual environment and activate it if one does not already exist.

This pvenv function is already configured system wide for fish on ExCL systems.

To create the virtual environment without using the wrapper function is also easy.

In bash:

In fish:

Here is the usage of venv which explains what the various flags do. From .

The virtual environment can be exited with deactivate.

Creating a Python Project in using the Hatch build system with CI support

provides a template for creating a python project using the hatch build system with CI support using ORNL's GitLab instance, complete with development documentation, linting, commit hooks, and editor configuration.

Steps to use the template:

2. Run setup_template.sh to setup the template for the new project.

3. Remove setup_template.sh

See for details on the template.

Using UV to create a python virtual environment with a specific version of python.

When a specific version of python is required, can be used to create a virtual environment with the specific version of python.

For example:

Use the command below to see the available python versions.

See and for details.

Visual Studio Code or VSCode is a lightweight but powerful source code editor which runs on your desktop and is available for Windows, macOS, and Linux. The editor has IntelliSense, debugger support, built-in git, and many extensions to add additional support to the editor. VSCode supports WSL and development on remote servers via ssh. Plugins add language support, linters, and compilers for many languages including Python, C/C++, CMake, and markdown.

Remote Explorer

The Remote - SSH and Remote - WSL are both extremely useful to edit code remotely on ExCL or locally in WSL if on a windows machine. Remote - SSH pulls the ssh targets from the users .ssh/config file. On Linux or MacOS, this process is straightforward and you likely already have an ssh config file setup. On Windows you have to specify the proxy command to use to proxy into the internal ExCL nodes. Here is an example file for Windows:

Here is the same file for Linux or MacOS:

The main difference between the files is that the Windows config has ProxyCommand with the windows ssh.exe and Linux and MacOS has ProxyJump, both commands setup the login node as a relay to the internal node.

Replace <Username> with your username. Other internal system can be added by copying the quad00 entry and modifying the name of the config and the HostName. It is highly recommended to use a passphrase protected ssh key as the login method. If you used a different name for the ssh key file, then replace ~/.ssh/id_rsa with your private key file. On Windows, this config file is located at %USERPROFILE%\.ssh\config. On Linux and MacOS, this config file is located at ~/.ssh/config. The config file doesn’t have an extension, but it is a text file that can be edited with vscode.

To avoid typing your ssh passphrase multiple times per login, use an SSH Agent to store the ssh credentials. See for details. On Windows, to enable SSH Agent automatically, start a local Administrator PowerShell and run the following commands:

On the ExCL side, you can add this code snippet to ~/.bashrc start the ssh-agent on login:

Important: Since VSCode installs its configuration to your home directory by default and the home directories are stored in NFS, the Remote.SSH: Lockfiles in Tmp setting needs to be checked. This setting is easiest to find with the settings search box.

The remote SSH explorer provides the same experience editing code remotely as you get when you are editing locally. Files that are opened are edited locally and saved to the remote server which helps when you have a slow connection to the remote which makes editing view vim and ssh too irresponsive. You can also access a remote terminal with ctl+`. The debuggers also run remotely. One gotcha is that extensions might need to be installed remotely for them to work properly. However, this is easy to do by clicking on the extension tab and choosing install local extensions on remote.

The ssh explorer also makes it easy to forward remote ports to the local machine. This is especially helpful when launching an http server or a jupyter notebook. See for details.

Debugging Using Run and Debug

Edit launch.json to define launch configurations according to the .

After generating a configuration from a template, the main attributes I add or change are "cwd" and "args". "args" has to be specified as an array, which is a pain. One workaround from suggests replacing " " with "," to avoid space separated arguments. For arguments with a value, "=" will need to be added between arguments and the value without spaces. When specifying "program" and "cwd" is is helpful to use the built in variables to reference the file or workspace folder. See .

Useful Extensions

• GrapeCity.gc-excelviewer

  • Preview CSV files.

• Gruntfuggly.todo-tree

Remote Work

• ms-vscode-remote.remote-containers

• ms-vscode-remote.remote-ssh

• ms-vscode-remote.remote-ssh-edit

Linters

• DavidAnson.vscode-markdownlint

  • Lint markdown files.

Language Support

• lextudio.restructuredtext

• ms-python.python

• ms-python.vscode-pylance

Git

• donjayamanne.githistory

• eamodio.gitlens

Note Taking/Knowledge Base

• foam.foam-vscode

Julia Language Extension

See .

Zenith 1

Created using PC Part Picker. The build is available at .

ExCl → User Documentation → Jupyter Quick Start

Installing Jupyter

Since there are many ways to install Jupyter using various python management tools, I will not reproduce the documentation here. The official documentation for installing Jupyter can be found at Project Jupyter | Installing Jupyter. However, I will highlight the methods of using Jupyter with UV, running Jupyter Notebooks in VS Code, and the alternative to Jupyter notebooks, Marimo | ExCL User Docs. These methods are all the methods that I typically use when working with python notebooks.

Jupyter with UV

See the UV documentation, . This documentation is well written and covers:

Jupyter kernels using virtual environments

See . Although , the following steps are still a good way to manually create and use a kernel from Jupyter.

Create a python virtual environment and activate it. Then install ipykernel and then install the kernel for use in Jupyter.

Use jupyter kernelspec list to view all the installed Jupyter kernels.

To uninstall a Jupyter kernel use uninstall.

Accessing a Jupyter Notebook Running on ExCL

A Jupyter notebook server running on ExCL can be accessed via a local web browser through port forwarding the Jupyter notebook's port. By default, this is port 8888 (or the next available port). This port might be in use if someone else is using running a notebook. You can specify the port with the --port flag when launching the Jupyter notebook. To use a different port just replace 8888 with the desired port number. In order to port forward from an internal node, you have to port forward twice, once from your machine to login.excl.ornl.gov and once again from the login node to the internal node (i.e. pcie).

Detailed instructions for Linux/Mac

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster. If you want to access a different system, then replace pcie with the system you intend to access.

1. Specify the ports that you intend to use. Choose a different number from the default so that you don't conflict with other users.

2. From your local machine connect to pcie using login.excl.ornl.gov as a proxy and local forward the jupyter port.

3. (Optional) Load the anaconda module if you don't have jupyter notebook installed locally.

If you ssh client is too old for proxyjump to work, you can always break up the process into another step.

1. From your local machine connect to login.excl.ornl.gov and local port forward port 8888.

2. From the login node connect to pcie and local port forward port 8888

3. Launch the Jupyter server on pcie

Detailed instructions for Windows with MobaXterm

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster.

1. From your local machine connect to login.excl.ornl.gov using MobaXterm.
2. Go to tools and click on MobaSSHTunnel. Use MobaSSHTunnel local forward port 8888.

   Click on MobaSSHTunnel

Detailed instructions for Windows with Visual Studio Code

These instructions go over how to access a Jupyter notebook running on the quad00 node in the ExCL Cluster using Visual Studio Code to handle port forwarding.

1. Open Visual Studio Code

2. Make sure you have the Remote - SSH extension installed.
3. Setup .ssh

These 2U servers are highly capable large memory servers, except that they have limited PCIe4 slots for expansion.

• HPE ProLiant DL385 Gen10 Plus chassis

• 2 AMD EPYC 7742 64-Core Processors

  • configured with two threads per core, so presents as 256 cores

  • this can be altered per request

• 1 TB physical memory

  • 16 DDR4 Synchronous Registered (Buffered) 3200 MHz 64 GiB DIMMS

• 2 HP EG001200JWJNQ 1.2 TB SAS 10500 RPM Disks

  • one is system disk, one available for research use

• 4 MO003200KWZQQ 3.2 TB NVME storage

  • available as needed

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM, and for networking/DPU research.

Status

Affirmed

Affirmed is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Justify

Justify is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Pharaoh

Pharaoh is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Secretariat

Secretariat is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Groq Links