This is the user documentation repository for the Experimental Computing Laboratory (ExCL) at Oak Ridge National Laboratory.

This site is undergoing development; systems and processes will be documented here as the documentation is created.

See the index on the left of this page for further detail.

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated. See Acknowledgment for details.

See Requesting access for information on how to request access to the system.

How to Login

See Access to ExCL for more details.

• Shell login: ssh login.excl.ornl.gov

• ThinLinc Session: https://login.excl.ornl.gov:300

Getting Assistance

Please send an email request to excl-help@ornl.gov for assistance. This initiates a service ticket and dispatches it to ExCL staff.

ExCL Cheat Sheet

Download Cheat Sheet

Please acknowledge in your publications the role the Experimental Computing Laboratory (ExCL) facility played in your research. Alerting us when a paper is accepted is also appreciated.

Sample acknowledgment:

This research used resources of the Experimental Computing Laboratory (ExCL) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725

You may use any variation on this theme, calling out specific simulations or portions of the research that used ExCL resources, or citing specific resources used.

However, the crucial elements to include are:

• The spelled out center name (it's okay to include the acronym, too): Experimental Computing Laboratory (ExCL)

• Office of Science and U.S. Department of Energy

• Contract No. DE-AC05-00OR22725

Additionally, when you add the paper to Resolution, please add “Experimental Computing Laboratory” to Research Centers and Institutes under Funding and Facilities as show in this image.

We appreciate your conscientiousness in this matter. Acknowledgment and pre-publication notification helps ExCL communicate the importance of its role in science to our sponsors and stakeholders, helping assure the continued availability of this valuable resource.

Two Nvidia H100s are now available on hudson.ftpn.ornl.gov. From Nvidia documentation:

The NVIDIA H100 NVL card is a dual-slot 10.5 inch PCI Express Gen5 card based on the NVIDIA Hopper™ architecture. It uses a passive heat sink for cooling, which requires system airflow to operate the card properly within its thermal limits. The NVIDIA H100 NVL operates unconstrained up to its maximum thermal design power (TDP) level of 400 W to accelerate applications that require the fastest computational speed and highest data throughput. The NVIDIA H100 NVL debuts the world’s highest PCIe card memory bandwidth of nearly 4,000 gigabytes per second (GBps)

Basic validation has been done via running the nvidia samples nbody program on both devices:

10485760 bodies, total time for 10 iterations: 401572.656 ms
= 2738.014 billion interactions per second
= 54760.284 single-precision GFLOP/s at 20 flops per interaction

The GPUs are available to the same UIDs as are using the A100s on milan0. If nvidia-smi does not work for you, you don't have the proper group memberships -- please send email to excl-help@ornl.gov and we will fix it. nvhpc is installed as a module as it is on other systems.

Description

EMU-Chick System is composed of 8x nodes that are connected via RapidIO Interconnect.

Each node has:

• 8x nodelets, array of DRAMs

• A stationary core (SC)

• Migration engine, PCI-Express interfaces, and an SSD.

• 64-byte channel 64GB of DRAM, divided into eight 8-byte narrow-channel-DRAMs (NC-DRAM

Each nodelet has:

• 2x Gosamer cores (GC)

• 64 concurrent in-order, single-issue hardware threads

Access

• The path to access to each individual EMU node is: login.excl.ornl.gov ⇒ emu-gw ⇒ emu ⇒ {n0-n7}

• emu-gw is an x86-based gateway node.

• The emu is the system board controller (sbc) and individual nodes are accessed only via this host.

• Connections to emu from the emu-gw are via preset ssh keys that are created during account creation. If you can't log in, your user account/project do not have access to EMU systems.

Development Workflow

• The EMU software development kit (SDK) is installed under /usr/local/emu on emu-gw, which is an x86 based system. Compilation and simulation should be performed on this machine.

• The official EMU programming guide is located under /usr/docs.

• emu and emu-gw mount home directories, so you should have no difficulty accessing your projects. Please use $HOME (or ${HOME}) as your home directory in scripts, as the mount location of your home directory, may change.

Other Resources

This document will be updated with additional documentation references and user information as it becomes available.

Contact

Please send assistance requests to excl-help@ornl.gov.

Description

This system is generally identical to the nodes (AC922 model 8335_GTW) in the ORNL OLCF Summit system. This system consists of

• 2 POWER9 (2.2 pvr 004e 1202) cpus, each with 22 cores and 4 threads per core.

• 6 Tesla V100-SXM2-16GB GPUs

• 606GiB memory

• automounted home directory (on group NFS server)

Contact

• excl-help@ornl.gov

Usage

As currently configured this system is usable using conventional ssh logins (from login.excl.ornl.gov), with automounted home directories. GPU access is currently cooperative; a scheduling mechanism and scheduled access is in design.

The software is as delivered by the vendor, and may not be satisfactory in all respects as of this writing. The intent is to provision a system that is as similar in all respects to Summit, but some progress is required to get there. This is to be considered an early access machine.

Please send assistance requests to excl-help@ornl.gov.

Installed Compilers

GPU Performance

This system is still being refined with respect to cooling. As of today, rather than running at the fully capable 300 watts per GPU, GPU usage has been limited to 250 watts to prevent overheating. As cooling is improved, this will be changed back to 300 watts with dynamic power reduction (with notification) as required to protect the equipment.

It is worth noting that this system had to be pushed quite hard (six independent nbody problems, plus CPU stressors on all but 8 threads) to trigger high temperature conditions. These limits may not be encountered in actual use.

Performance Information

GPU performance information can be viewed at

Request access by emailing excl-help@ornl.gov.

Other Resources

Currently has a U250 installed with a custom application deployed which requires an older linux kernel.

Lewis is configured with kernel 5.15.0.

Hold set with:

To remove hold:

pcie

Description

This system is intended for pci-based device support.

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Atipa

• Tyan Motherboard S7119GMR-06

• 192 GB memory

• Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHzIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz 2x16 cores no hyperthreading

• Centos

Use

This system is used for heterogeneous accelerator exploration and FPGA Alveo/Vitis-based development.

Current VMs

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to excl-help@ornl.gov.

These 2U servers are highly capable large memory servers, except that they have limited PCIe4 slots for expansion.

• HPE ProLiant DL385 Gen10 Plus chassis

• 2 AMD EPYC 7742 64-Core Processors

  • configured with two threads per core, so presents as 256 cores

  • this can be altered per request

• 1 TB physical memory

  • 16 DDR4 Synchronous Registered (Buffered) 3200 MHz 64 GiB DIMMS

• 2 HP EG001200JWJNQ 1.2 TB SAS 10500 RPM Disks

  • one is system disk, one available for research use

• 4 MO003200KWZQQ 3.2 TB NVME storage

  • available as needed

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM, and for networking/DPU research.

Status

Affirmed

Affirmed is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Justify

Justify is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Pharaoh

Pharaoh is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Centos 7.9.

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

Secretariat

Secretariat is one of our triple crown servers (named after Triple Crown winners). These are highly capable large memory servers

It currently runs Ubuntu 22.04.

Specialized hardware

• BlueField-2 DPU connected to 100Gb Infiniband Network

  • Can also be connected to 10Gb ethernet network

  • used to investigate properties and usage of the NVidia BlueField-2 card (ConnectX-6 VPI with DPU).

Usage

These servers are generally used for customized VM environments, which are often scheduled via SLURM.

The Experimental Computing Laboratory is a Advanced Computing Systems Research project directed by Jeffrey Vetter. Support staff include:

• Steve Moulton - systems engineer

• Aaron Young - software engineer

Contact excl-help@ornl.gov for assistance.

To become authorized to access ExCL facilities, please apply at https://www.excl.ornl.gov/accessing-excl/. You have the option of using your ORNL (ucams) account if you have one, or creating an xcams (external user) account if you wish.

Once you have access you have a couple of options.

• login.excl.ornl.gov runs an SSH Server and you can connect to the login node with ssh login.excl.ornl.gov.

• There is a limited number of ThinLinc licenses available. Thinlinc (Xfce Desktop) can be accessed at https://login.excl.ornl.gov:300 for HTML5 services, and ThinLinc clients can use login.excl.ornl.gov as their destination. ThinLinc clients can be downloaded without cost from https://www.cendio.com/thinlinc/download. ThinLinc provides much better performance than tunneling X over SSH. A common strategy is to access login.excl.ornl.gov via ThinLinc and then use X11 forwarding to access GUIs running on other nodes.

Notes:

• Using an SSH key instead of a password to connect to ExCL is highly recommended. See How to get start with SSH keys. SSH keys are more secure than passwords, and you are less likely to accidentally get banned from multiple incorrect login attempts when using SSH Keys to authenticate. If you get blocked, you can send a help ticket to excl-help@ornl.gov with your IP address to get removed from the block list.

• If you use a passphrase with your SSH key (recommended for security), you should also set up an SSH Agent to load the SSH key. An SSH Agent allows you to enter your passphrase once for the session without needing to enter your passphrase many times. The VS Code documentation is well written for setting up this SSH Agent on a variety of platforms; see Visual Studio Code Remote Development Troubleshooting Tips and Tricks.

• It is recommended to use a terminal multiplexer like tmux or screen. These tools keep your session active and can be reattached to if you loose network connection. They also allow you to open multiple windows or split panels.

Add SSH Public Key to ExCL’s Authorized Keys

You can manually copy the key if already on ExCL. For example

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Or you can you ssh-copy-id to copy your local systems key to ExCL.

ssh-copy-id login.excl.ornl.gov

ExCL reserves the first Tuesday of every month for systems maintenance. This may result in complete inaccessibility during business hours. Every effort will be made to minimize the scope, duration, and effect of maintenance activities.

If an outage will affect urgent projects (i.e., with impending deadlines) please email excl-help@ornl.gov as soon as possible.

ExCL Server List with Accelerators

New Systems and Devices to be Deployed

• 2 Snapdragon HDK & Display

• Intel ARC GPU

• Achronix FPGA

• AGX Orin Developer Kits

• Xilinx U280

Accelerator Highlights

Unique Architecture Highlights

Other Equipment

• RTP164 High Performance Oscilloscope

Primary Usage Notes

Access Host (Login)

Login is the node use to access ExCL and to proxy into and out of the worker nodes. It is not to be used for computation but for accessing the compute notes. The login node does have ThinLinc installed and can also be used for graphical access and more performance x11 forwarding from an internal node. See ThinLinc Quickstart.

General Interactive Login Use

These nodes can be access with ssh, and are availible for general interactive use.

Notes:

• All of the general compute resources have hyperthreading enabled unless otherwise stated.. This can be changed on a per request basis.

• TL: Thinlinc enabled. Need to use login as a jump host for resources other than login. See ThinLinc Quickstart

• Slurm: Node is added to a slurm partition and will likely be used for running slurm jobs. Try to make sure your interactive use does not conflict with any active Slurm jobs.

  • Most of the general compute resources are Slurm-enabled, to allow queuing of larger-scale workloads. excl-help@ornl.gov for specialized assistance. Only the systems that are heavily used for running Slurm jobs are marked “Slurm” above.

Graphical session use via ThinLinc

• login — not for heavy computation

• zenith

• zenith2

• clark

• lewis

• pcie

• intrepid

• spike

Slurm for Large Jobs

• Triple Crown — Dedicated Slurm runners.

  • affirmed

  • justify

  • secretariat

  • pharaoh

• Milan — Additional Slurm Resources with other shared use.

  • milan0

  • milan1

  • milan3

• Others — Shared slurm runners with interactive use.

  • milan[0-3]

  • cousteau

  • excl-us03

  • explorer

  • oswald

  • oswald[00, 02-03]

Gitlab Runner Speciliazed Nodes

• slurm-gitlab-runner — Gitlab Runner for launching slurm jobs.

• docker — for docker runner jobs.

• devdoc — for internal development documentation building and hosting.

Note: any node can be used as a CI runner on request. See GitLab Runner Quickstart and GitHub Runner Quickstart. The above systems have a dedicated or specilized use with CI.

Docker

• docker — Node with docker installed.

Specialized usage and reservations

Notes:

• task-reserved: reserved for specialized tasks, not for project

Infrastructure Systems

Description

The Qualcomm® Snapdragon™ 855 Mobile Hardware Development Kit (HDK) is a highly integrated and optimized Android development platform.

Accessing this system:

• Qualcomm board is connected to an HPZ820 workstation (McMurdo) or to an HP Z4 workstation (Clark) through USB

• Development Environment: Android SDK/NDK

• Login to mcmurdo or clark

  • $ ssh –Y mcmurdo

• Setup Android platform tools and development environment

  • $ source /home/nqx/setup_android.source

• Make sure you have a functining environment

  • adb kill-server

  • adb start-server

  • adb root (restart adbd as root)

  • adb devices (to make sure there is a snapdragon responding)

  • adb shell (to test connecting to the device)

• Run Hello-world on ARM cores

  • $ make compile push run

• Run OpenCL example on GPU

  • Run Sobel edge detection

    • $ make compile push run fetch

  • Login to Qualcomm development board shell

    • $ adb shell

    • $ cd /data/local/tmp

Other Details

The snapdragon SDK uses python 2.7; you may need to explicitly specify python2 in your environment.

Access

Access will be granted per request (as this cannot be used as a shared resource).

Useful Links

Images

oswald00

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

• Centos

Access

There is not currently special access permissions. System is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to excl-help@ornl.gov.

oswald01

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

• Centosa

• Micron 9100 NVM 2.4TB MTFDHAX214MCF

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to excl-help@ornl.gov.

oswald02

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

• Centos

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to excl-help@ornl.gov.

oswald03

Description

This system is a generic development server purchased with the intent of housing various development boards as needed.

The system is

• Penguin Computing Relion 2903GT

• Gigabyte motherboard MD90-FS0-ZB

• 256 GB memory

• Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 2x16 cores no hyperthreading

• Centos

Access

There is not currently special access permissions. The system is available to ExCL users. This may change as needed.

Images

Contact

Please send assistance requests to excl-help@ornl.gov.

Apptainer/Singularity is the most widely used container system for HPC. It is designed to execute applications at bare-metal performance while being secure, portable, and 100% reproducible. Apptainer is an open-source project with a friendly community of developers and users. The user base continues to expand, with Apptainer/Singularity now used across industry and academia in many areas of work.

Apptainer is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.

Apptainer allows for more secure containers than docker without the need for root access.

Why use Apptainer?

From Why you should use Apptainer vs Docker | Medium.

Apptainer allows you to:

1. Build on a personal computer with root or on a shared system with fakeroot.

2. Move images between systems easily.

3. Execute on a shared system without root.

Apptainer is designed for HPC:

1. Defaults to running as the current user

2. Defaults to mounting the home directory in /home/$USER

3. Defaults to running as a program (not background process)

Apptainer also has great support with Docker images.

Systems with Apptainer installed

• docker

• thunderx

• zenith

Other systems can have Apptainer installed by request.

Notes:

• Apptainer mounts $HOME , /sys:/sys , /proc:/proc, /tmp:/tmp, /var/tmp:/var/tmp, /etc/resolv.conf:/etc/resolv.conf, /etc/passwd:/etc/passwd, and $PWD by default and run in ~ by default. This means you can change files in your home directory by running with Apptainer. This is different from Docker which creates a container (overlay in Apptainer) by default for the application to run in. See Bind Paths and Mounts.

• To mount another location when running Apptainer, use the --bind option. For example to mount /noback use --bind /noback:/noback. See Bind Paths and Mounts.

• Admins can specify default bind points in /etc/apptainer/apptainer.conf. See Apptainer Configuration Files

• When creating a definition file, pay attention to the rules for each section. See Definition Files For example:

  • %setup is a scriplet which runs outside the container and can modify the host. Use ${APPTAINER_ROOTFS} to access the files in the Apptainer image.

  • Environment variables defined in %environment are available only after the build, so if you need access to them for the build, define them in the %post section.

• To use --fakeroot you must first have fakeroot configured for that user. This can be done with the command sudo apptainer config fakeroot --add <user>. See User Namespaces & Fakeroot

• To use X11 applications in Apptainer with over ThinLinc, you need to bind /var/opt/thinlinc with --bind /var/opt/thinlinc since that is where the user’s XAuthority file is stored.

• sandbox image build mode along with fakeroot can help if one needs to apt-get install or yum install packages within a singularity / apptainer container and persist the mutable image out on disk: Build a Container — Apptainer User Guide main documentation.

NFS Limitations

From https://apptainer.org/docs/admin/main/installation.html#nfs.

NFS filesystems support overlay mounts as a lowerdir only, and do not support user-namespace (sub)uid/gid mapping.

• Containers run from SIF files located on an NFS filesystem do not have restrictions.

• In setuid mode, you cannot use --overlay mynfsdir/ to overlay a directory onto a container when the overlay (upperdir) directory is on an NFS filesystem. In non-setuid mode and fuse-overlayfs it is allowed but will be read-only.

• When using --fakeroot and /etc/subuid mappings to build or run a container, your TMPDIR / APPTAINER_TMPDIR should not be set to an NFS location.

• You should not run a sandbox container with --fakeroot and /etc/subuid mappings from an NFS location.

Getting Started

• Apptainer—Documentation

• Apptainer—Quickstart

• Apptainer—Support for Docker and OCI Containers

• Apptainer—Running Services

• Apptainer—CLI Run

• Apptainer—Bind Paths and Mounts

• Apptainer—Definition Files

Apptainer with Harbor

See registry (ornl.gov) for general information for how to use the ORNL Container Repositories. These sites https://camden.ornl.gov and https://savannah.ornl.gov are the internal and external container repositories running Harbor.

These container registry also work with Apptainer images.

Follow the regular instructions to setup Harbor. Then see the commands below for an Apptainer specific reference.

Login to Camden

apptainer registry login -u ${email_address} oras://camden.ornl.gov

Logout of Camden

apptainer registry logout oras://camden.ornl.gov

Pull image from Camden

apptainer pull <myimage>.sif oras://camden.ornl.gov/<myproject>/<myimage>[:<tag>]

Push image to Camden

apptainer push <myimage>.sif oras://camden.ornl.gov/<myproject>/<myimage>[:<tag>]

CI with Apptainer and Harbor

Create a robot account in Harbor using the regular method.

Then use the CI environment variables APPTAINER_DOCKER_USERNAME and APPTAINER_DOCKER_PASSWORD to specify the robot username and token. Make sure to deselect Expand variable reference since the username has a ‘$’ in it.

System Admin Notes

• It is helpful to add commonly needed bind paths to /etc/apptainer/apptainer.conf. I have added the following bind commands to Zenith:

bind path = /scratch
bind path = /etc/localtime
bind path = /etc/hosts
bind path = /var/opt/thinlinc
bind path = /auto

See also

• ORNL uses can also look at this ornl-containers / singularity page for more details on using containers at ORNL.

I can't log in

There are two most likely sources of this problem

Password won't work

The most frequent cause is having your visitor (non-ORNL internal password) wrong, or having had it expire. See https://xcams.ornl.gov to address this. If you are ORNL staff, a frequent cause is a failure to keep your internal ORNL systems password up to date (UCAMS) or have missed required training. ExCL makes the same check that any ORNL system makes as to whether a password is valid or an account exists (you will not be able to differentiate the two errors based on the login failure). This will look like

$ ssh login.excl.ornl.gov
hsm@login.excl.ornl.gov's password:
Permission denied, please try again.

Too many login attempts from a given IP address

ExCL limits logins to five consecutive failures within a short period of time. After that limit exceeded, login atempts from your IP address will be blocked. This might look like

$ ssh login.excl.ornl.gov
ssh: connect to host login.excl.ornl.gov port 22: Operation timed out

To have this addressed, report your IP address to excl-help@ornl.gov. If you are on an ORNL network, you can use the usual native tools on your system to find your IP address. If you are at home and on a network using NAT (as most home networks do) use What Is My IP? Best Way To Check Your Public IP Address to determine your public IPv4 address when external to the lab. Note that this will not report the correct address if you are on an ORNL (workstations or visitor) network.

I can’t clone my git repo

The recommended approach for accessing git repositories in ExCL is to use the SSH protocol instead of the HTTPS protocol for private repositories and either protocol for public repositories. However, both approaches will work with the proper proxies, keys, applications passwords, and password managers in place.

To use the SSH protocol you must first setup SSH keys to the git website (i.e. GitLab, GitHub, and Bitbucket). See Git - Setup Git access to code.ornl.gov | ExCL User Docs (ornl.gov) for details for how to do this for code.ornl.gov. The other Git Clouds have similar methods to add SSH keys to your profile.

Since the worker nodes are behind a proxy. You must setup an SSH jump host in your .ssh/config to access Git SSH servers. See Git - Git SSH Access | ExCL User Docs (ornl.gov) to verify that you have setup the proper lines in your SSH Config.

I need a newer version of pip

See Python | ExCL User Docs for instructions on how to setup a Python virtual environment with the latest version of pip.

I need a newer version of Python

See Python | ExCL User Docs for instructions on how to use UV to setup a Python virtual environment with a specific python version.

ExCl → User Documentation → Contributing

About ExCl User Documentation

Documentation published to ExCL users is available in our GitHub repo. Users are encouraged to contribute by improving the material or providing user-created tutorials to share with the community.

Ways to Contribute

Would you like to make things better? There are a few ways you can contribute to improving our documentation and adding user-created tutorials or content.

1. Email your suggestions to the team excl-help@ornl.gov

2. Want to change things? Feeling adventurous? Comfortable with git? See instructions for our Git workflow to branch our documentation repository and hack away. You got this.

Roadmap for Setup

If you are new to remote development on ExCL here is a roadmap to follow to set important settings and to get familiar with remote Linux development.

1. Access ExCL

2. Setup SSH: SSH Keys for Authentication | ExCL User Docs

   • Bonus: SSH-Agent and SSH Forwarding

3. Setup Git

   1. Git SSH Access | ExCL User Docs

   2. Setup Git access to code.ornl.gov | ExCL User Docs

4. Setup VS Code Remote Explorer: Visual Studio Code Remote Explorer | ExCL User Docs

   • Important: Make sure to check the setting Remote.SSH: Lockfiles in Tmp.

5. Setup FoxyProxy. This enables access to ThinLinc as well as any other web services running on ExCL systems.

6. Now you are ready to follow any of the other Quick-Start Guides.

Setup FoxyProxy

1. Launch SOCKS dynamic proxy forwarding to the login node using dynamic forwarding with SSH. On Linux or macOS, via the SSH flag -D

   Copy

    $ ssh -D 9090 <Username>@login.excl.ornl.gov

   or in the ssh config add the DynamicForward option

   Copy

   DynamicForward 9090

   On Windows, use MobaSSHTunnel to set up Dynamic Forwarding. See Jupyter Quickstart for more information on port forwarding in windows.

2. Setup FoxyProxy Install the FoxyProxy Chrome extension or Firefox extension.

   Setup FoxyProxy by adding a new proxy for localhost on port 9090. Then add the regular expression URL pattern .*\.ftpn\.ornl\.gov to forward ThinLinc traffic to ExCL.

Zenith 1

Created using PC Part Picker. The build is available at https://pcpartpicker.com/list/xPkRwc.

PCPartPicker Part List

Access to the GPUs

To have access to the GPUs, request to be added to the video and render groups if you are not already in these groups.

Images

Zenith 2

Created using PC Part Picker. The build is available at https://pcpartpicker.com/list/vjXBPF.

PCPartPicker Part List

Images

Backup

Snapshots take space for files that have changed or been deleted. They are automatically deleted as they age, so that hourlies are kept for 48 hours, one hourly from each day is kept for 30 days, and one hourly for each 30 day period is kept for 180 days. This policy can be modified on request. Snapshots are read only; you can copy files from them back into your home directory tree to restore them.

There is currently no file purge policy. Given that ExCL researchers take care of cleaning up files that are no longer in use, no change to this policy is foreseen. Files for inactive users are archived in a non-snapshot file system. While it is our intent to continue maintaining storage for inactive users, this policy may change in the future.

Quotas

Local Storage

Project Storage

Shared storage space for collaborative projects is available upon request. Each project is assigned a dedicated subvolume within the ZFS filesystem, which is accessible via an automounted NFS share. The mount point for each project is located at:

Access to the project directories is restricted for security and organization. Only execute permissions are set on the /auto/projects/ directory, meaning you must know the specific project name to cd into it. You will not be able to use ls to list all available project directories.

Access Control Lists (ACLs) are used to manage permissions for project directories, allowing for flexible access configurations. By default, all members associated with a project will have read, write, and execute permissions for the files within their assigned project directory.

DevDocs

The documentation for hunter is built with GitLab-CI. Here are the relevant lines in .gitlab-ci.yml.

Request a DevDocs Site

• URL

• Project Name (This will be your DevDocs subdirectory)

Use module load julia to load the Julia tooling on an ExCL system.

Julia VSCode Extension in ExCL

This can be done by setting the julia.executablePath to point to the Julia executable that the extension should use, which is this case is the one loaded by the module load command for the version of Julia you want to use. Once set, the extension will always use that version of Julia. To edit your configuration settings, execute the Preferences: Open User Settings command (you can also access it via the menu File->Preferences->Settings), and then make sure your user settings include the julia.executablePath setting. The format of the string should follow your platform specific conventions, and be aware that the backlash \ is the escape character in JSON, so you need to use \\ as the path separator character on Windows.

To find the proper path to Julia, you can use which julia after the module load command. At the time of writing this page, the default version of Julia installed on ExCL is 1.10.4 and the julia.executablePath should be set as shown below.

Using Julia with Jupyter

Within ExCL, the first step is to load the Julia module with module load julia to load the Julia tooling into the ExCL system.

The third step is to install ‘IJulia’ using the Julia REPL. Launch the Julia REPL with julia then press ] to open the package management, then run add IJulia.

Installing Conda

Follow the prompts on the installer screens. Accept the license agreements. Specify /home/$USER/conda as the installation location. Choose if you want the installer to initialize Miniconda.

Improving Conda Environment Solver Performance

The quick start guide is below.

Installing Spack

Installing Jupyter

Jupyter with UV

Jupyter kernels using virtual environments

Create a python virtual environment and activate it. Then install ipykernel and then install the kernel for use in Jupyter.

Use jupyter kernelspec list to view all the installed Jupyter kernels.

To uninstall a Jupyter kernel use uninstall.

Accessing a Jupyter Notebook Running on ExCL

A Jupyter notebook server running on ExCL can be accessed via a local web browser through port forwarding the Jupyter notebook's port. By default, this is port 8888 (or the next available port). This port might be in use if someone else is using running a notebook. You can specify the port with the --port flag when launching the Jupyter notebook. To use a different port just replace 8888 with the desired port number. In order to port forward from an internal node, you have to port forward twice, once from your machine to login.excl.ornl.gov and once again from the login node to the internal node (i.e. pcie).

Detailed instructions for Linux/Mac

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster. If you want to access a different system, then replace pcie with the system you intend to access.

1. Specify the ports that you intend to use. Choose a different number from the default so that you don't conflict with other users.
2. From your local machine connect to pcie using login.excl.ornl.gov as a proxy and local forward the jupyter port.
3. (Optional) Load the anaconda module if you don't have jupyter notebook installed locally.
4. Launch the Jupyter server on pcie
5. Connect to the Jupyter notebook using a web browser on your local machine. Use the token shown in the output from running the Jupyter server. Url: http://localhost:<local_port>/?token=<token>. You can also configure jupyter to use a password with jupyter notebook password if you don't want to use the access tokens.

If you ssh client is too old for proxyjump to work, you can always break up the process into another step.

1. From your local machine connect to login.excl.ornl.gov and local port forward port 8888.
2. From the login node connect to pcie and local port forward port 8888
3. Launch the Jupyter server on pcie
4. Connect to the Jupyter notebook using a web browser on your local machine. Use the token shown in the output from running the Jupyter server. Url: http://localhost:8888/?token=<token>

Detailed instructions for Windows with MobaXterm

These instructions go over how to access a Jupyter notebook running on the pcie node in the ExCL Cluster.

1. From your local machine connect to login.excl.ornl.gov using MobaXterm.
2. Go to tools and click on MobaSSHTunnel. Use MobaSSHTunnel local forward port 8888.

   Click on MobaSSHTunnel

   Click on New SSH Tunnel

   Local port forward 8888

   Click the play button to start port forwarding

3. From the login node connect to pcie and local port forward port 8888
4. Launch the Jupyter server on pcie
5. Connect to the Jupyter notebook using a web browser on your local machine. Use the token shown in the output from running the Jupyter server. URL: http://localhost:8888/?token=<token>

Detailed instructions for Windows with Visual Studio Code

These instructions go over how to access a Jupyter notebook running on the quad00 node in the ExCL Cluster using Visual Studio Code to handle port forwarding.

1. Open Visual Studio Code

2. Make sure you have the Remote - SSH extension installed.
3. Setup .ssh

   Navigate to the remote explorer settings.

   Chose the user .ssh config.

   Add the remote systems to connect to with the proxy command to connect through the login node.

4. Connect to the remote system and open the Jupyter folder.

   Open Folder

5. Run the Jupyter notebook using the built-in terminal.
6. Open the automatically forwarded port.

Register a Runner

Runners can be registered as either a group runner or for a single repository (also know as a project runner). Group runners are are made available to all the repositories in a group.

• URL

• Registration Token

• Executor (choose shell or docker with image)

• Project Name (This can be group name or repo name)

• ExCL System

• Tag List

After the runner is added, you can edit the runner to change the tags and description.

Group Runner

Single Repo Runner (Project Runner)

List of ExCL Systems with a runner

Any system can be requested as a runner. These systems are already being used as a runner. (Updated October 2023)

• docker.ftpn.ornl.gov

• explorer.ftpn.ornl.gov

• intrepid.ftpn.ornl.gov

• justify.ftpn.ornl.gov

• leconte.ftpn.ornl.gov

• lewis.ftpn.ornl.gov

• milan2.ftpn.ornl.gov

• milan3.ftpn.ornl.gov

• oswald00.ftpn.ornl.gov

• oswald02.ftpn.ornl.gov

• oswald03.ftpn.ornl.gov

• pcie.ftpn.ornl.gov

• zenith.ftpn.ornl.gov

Using Slurm with Gitlab CI

The system slurm-gitlab-runner is setup specifically to run CI jobs that then run the execution using slurm with sbatch --wait.

This template includes two helper scripts, runner_watcher.sh and slurm-tee.py.

runner_watcher.sh watches the CI job and cancels the Slurm job if the CI job is canceled or times out.

slurm-tee.py watches the slurm-out.txt and slurm-err.txt files and prints their content to std-out so that the build log can be watched from the GitLab web interface. Unlike regular less --folow, slurm-tee watches the multiple files for changes and also exits once the slurm job completes.

Spack and Conda with Gitlab-CI

Groq Links

Login and Groq Cards Available

Start by logging into ExCL's login node.

From the login node, you can then login to a node with a Groq card, for example

Here is a table of the Groq cards available:

Groq card and Slurm

The recommended way to access the Groq card is to reserve it through the Slurm resource manager. Groq cards are available on machines in the groq partition. To reserve a node with a groq card for interactive use use the command.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

Non-interactive batch jobs can similarly be launched.

Where: -J, --job-name=<jobname> specifies the job name. -p, --partition=<partition names> specifies the partition name. --exclusive specifies you want exclusive access to the node. --gres="groq:card:1" specifies that you want to use 1 groq card.

or specified in the script:

Using the Groq Card

In order to use the Groq API you need to make sure you are using python 3.8 and that you add the Groq python libraries to your path. For python 3.8 you can either use the installed system python3.8 or use conda to install python3.8.

System python3.8

You need to fully quantify the path to python since Ubuntu 22.04 defaults to python3.10. This means you need to use

Then to install jupyter notebook in your home directory, you would need to do

Conda

Graphical Access to Groq Systems

Jupyter Notebooks

Getting started Resources

Useful Groq Commands

• Run regression tests to verify card functionality: /opt/groq/runtime/site-packages/bin/tsp-regression run

• Get Groq device status: /opt/groq/runtime/site-packages/bin/tsp-ctl status

• Monitor temperature and power: /opt/groq/runtime/site-packages/bin/tsp-ctl monitor

The EDA tools are installed on the system dragon. dragon can be access via ssh from the login node, via x11 forwarding from the login node's ThinLink, or directly via ThinLink with Foxy Proxy. See ThinLinc Quickstart to get started with ThinLinc Setup. See Accessing ExCL for more details on logging in.

Ssh access:

ssh -Y -J <username>@login.excl.ornl.gov <username>@dragon

ThinLinc access to login:

https://login.excl.ornl.gov:300

ThinLinc access to dragon (Requires reverse proxy to be setup):

https://dragon.ftpn.ornl.gov:300

All of the tools are installed to /opt/Siemens and the tools can be set up with

source /opt/Siemens/setup.sh

Also, please join the siemens-eda slack channel in the ORNL CCSD slack.

Compilers are, in general, maintained from a central NFS repository, and made accessible via the module command (from Lmod). For example

hsm@secretariat:~$ module load gnu
hsm@secretariat:~$ module avail

---------------- /usr/share/lmod/lmod/modulefiles ----------------
   Core/lmod/6.6    Core/settarg/6.6

------ /auto/software/swtree/ubuntu20.04/x86_64/modulefiles ------
   anaconda/3          git/2.38.0             julia/1.8.0
   cmake/3.22.5        gnu/10.2.0             llvm/8.0.1
   gcc/10.2.0          gnu/11.1.0             llvm/13.0.1
   gcc/11.1.0          gnu/11.3.0             llvm/14.0.0 (D)
   gcc/11.3.0          gnu/12.1.0    (L,D)
   gcc/12.1.0   (D)    hipsycl/0.9.2

  Where:
   L:  Module is loaded
   D:  Default Module

If you do not load a module, you will get the default compiler as delivered by the operating system vendor (4.8.5 on some systems). If you module load gnu you will currently get 12.1.0, as it is the default. If you need, say, 10.2.0, you need to module load gnu/10.2.0. Note that documentation details with respect to compiler availability and versions will not necessarily be kept up to date; the system itself is authoritative.

Some compilers (notably xlc and the nvhpc tool chain) cannot be installed on nfs, so if they are availble they will show up in a different module directory. The same module commands are used.

Additional compilers can be installed on request to excl-help@ornl.gov. Maintaining multiple Gnu suites is straightforward, less so for other tool suites.

Additional compilers and tools can also be installed using Spack.

Link: Open WebUI (Running on Zenith) Website: Open WebUI Documentation: 🏡 Home | Open WebUI GitHub: open-webui/open-webui: User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

There is an Open WebUI server running on ExCL for developing and testing LLM models created with Ollama. In order to use the website you must first Setup FoxyProxy, then the above link will work. When you first access the page, you will be prompted to create a new account. This account is a unique account for this instance of Open WebUI and is not tied to anything else. After creating an account, send a message to Aaron Young or excl-help@ornl.gov to request your account to be ungraded to an admin account.

Ollama is deployed in ExCL as a module. To use Ollama, load the module, and then you have access to the ollama CLI interface.

Load the Ollama module with:

module load ollama

Ollama has a server component which stores files in its home. This server component should be launched using a service account by ExCL admin, since it provides ollama for the entire system. Ollama is already running on some of the workers in ExCL. See the output from the model load for an up-to-date list. Contact excl-help@ornl.gov if you would like ollama to be available on a specific system.

Ollama API

Examples of using the Ollama API can be found at ollama-python/examples/chat.py.

Links

• Ollama Website

• Ollama GitHub

• Ollama CLI Reference

This page covers a few recommendations and tips for getting started with Python in ExCL following best practices for packaging python projects and using virtual environments. There are many different ways to structure and package python projects and various tools that work with python, so this page is not meant to be comprehensive but to provide a few recommendations for getting started.

Python Virtual Environments with venv

Using virtual environments is the recommended way to isolate Python dependencies and ensure compatibility across different projects. Virtual environments prevent conflicts between packages required by different projects and simplify dependency management. The goal with isolated, project specific python environments is to avoid the situation found in https://xkcd.com/1987/.

If you are a using the fish shell, the simple function show below is a wrapper around venv to activate a python virtual environment if one already exists in .venv in the current directory or create a new virtual environment and activate it if one does not already exist.

function pvenv --wraps='python3 -m venv --upgrade-deps venv' --description 'Create and activate a python virtual environment in .venv with updated pip and prompt set to the folder\'s name'
   if test -e .venv/bin/activate.fish
      echo Using existing `.venv`.
      source .venv/bin/activate.fish
   else
      echo Creating new `.venv`.
      python3 -m venv --upgrade-deps --prompt (basename $PWD) .venv $argv; and source .venv/bin/activate.fish;
   end
end

This pvenv function is already configured system wide for fish on ExCL systems.

To create the virtual environment without using the wrapper function is also easy.

In bash:

python3 -m venv --upgrade-deps --prompt $(basename $PWD) .venv
source .venv/bin/activate

In fish:

python3 -m venv --upgrade-deps --prompt (basename $PWD) .venv
source .venv/bin/activate.fish

Here is the usage of venv which explains what the various flags do. From venv — Creation of virtual environments — Python 3.13.1 documentation.

usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
            [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps]
            [--without-scm-ignore-files]
            ENV_DIR [ENV_DIR ...]

Creates virtual Python environments in one or more target directories.

positional arguments:
  ENV_DIR               A directory to create the environment in.

options:
  -h, --help            show this help message and exit
  --system-site-packages
                        Give the virtual environment access to the system
                        site-packages dir.
  --symlinks            Try to use symlinks rather than copies, when
                        symlinks are not the default for the platform.
  --copies              Try to use copies rather than symlinks, even when
                        symlinks are the default for the platform.
  --clear               Delete the contents of the environment directory
                        if it already exists, before environment creation.
  --upgrade             Upgrade the environment directory to use this
                        version of Python, assuming Python has been
                        upgraded in-place.
  --without-pip         Skips installing or upgrading pip in the virtual
                        environment (pip is bootstrapped by default)
  --prompt PROMPT       Provides an alternative prompt prefix for this
                        environment.
  --upgrade-deps        Upgrade core dependencies (pip) to the latest
                        version in PyPI
  --without-scm-ignore-files
                        Skips adding SCM ignore files to the environment
                        directory (Git is supported by default).

Once an environment has been created, you may wish to activate it, e.g. by
sourcing an activate script in its bin directory.

The virtual environment can be exited with deactivate.

Creating a Python Project in using the Hatch build system with CI support

Python Project Template provides a template for creating a python project using the hatch build system with CI support using ORNL's GitLab instance, complete with development documentation, linting, commit hooks, and editor configuration.

Steps to use the template:

1. Fork the respository.

2. Run setup_template.sh to setup the template for the new project.

3. Remove setup_template.sh

See Python Project Template Documentation for details on the template.

Using UV to create a python virtual environment with a specific version of python.

When a specific version of python is required, uv can be used to create a virtual environment with the specific version of python.

uv venv --python <version>

For example:

uv venv --python 3.11

Use the command below to see the available python versions.

uv python list

See astral-sh/uv - python management and uv docs - installing a specific version for details.

The login node has ThinLinc install and can be accessed at https://login.excl.ornl.gov:300. Since this node is public facing, it is the easiest to access with ThinLinc.

In addition to the login node, multiple systems including the virtual systems have ThinLinc installed, which makes it easier to run graphical applications. To access ThinLinc you need to use as socks proxy to forward traffic to the ExCL network or port forwarding of port 22 to use the ThinLinc client.

For better keyboard shortcut support and to prevent the browser from triggering the shortcuts, I recommend installing Open-as-Popup.

Systems Availible

Accessing ThinLinc through the web interface

1. Setup FoxyProxy and make sure to have the SOCKS dynamic proxy running.

2. Connect to the ThinLinc server using the links above.

Accessing ThinLinc through ThinLinc Client

This approach is recommended if you need better keyboard forwarding support for keyboard shortcuts that are not working with the Web client. The web client approach is easier to use and enables connecting to multiple systems at a time.

If the system is directly accessible (for example login.excl.ornl.gov), then you can specify the system and connect directly.

If the system is an internal node, then local port forwarding must be used. The steps to setting this up are as follows.

1. Forward port 22 from the remote system to your local system through login. On Linux or macOS

   Copy

    $ ssh -L <localport>:<hostname>:22 <Username>@login.excl.ornl.gov

   On windows use ssh via powershell, MobaSSHTunnel, Visual Studio Code, or putty to forward port 22. See Jupyter Quickstart for more information on port forwarding in windows.

2. Add alias in hosts file for the remote node. This is needed because of how ThinLinc establishes the remote connected. On Linux this host file is /etc/hosts. On windows the file is C:\Windows\System32\drivers\etc\hosts. Host file:

   Copy

   127.0.0.1 <hostname>
   ::1       <hostname>

3. Launch the ThinLinc Client.

4. In the options, specify the SSH port to be <localport>.

5. Specify the Server, Username, and credentials.

6. Connect to the server with "Connect".

Potential Issues you may encounter

If you use Gnome and do not have access to the module command when you start a terminal session over ThinLinc web, then your terminal session may not be configured as a login session. To resolve

1. Right click on the terminal icon on the left side of your screen

2. In Preferences -> Unnamed, make sure Run command as a login shell is checked.

You will then get login processing (including sourcing the /etc/profiles.d files) and so the module command will now be present.

Thank you Chen Zhang for the presentation materials to learn about and get started with Marimo. Marimo works well in ExCL and can be set up to work with the Ollama instance running in ExCL to enable the AI features.

Download Marimo Quick-start Presentation

Setup Runner as a service in ExCL

If you do want to register the runner as a service, the easiest way is to use systemd user services. To set this up follow the steps below.

Create a user systemd config which is unique to a single system.

Notes:

• If you setting up a second runner, the ln command will fail if the link already exists. Ensure that the link is a valid link pointing to scratch before continuing with these instructions.

Create a folder to store the GitHub Runner.

~/github-runners/<node>-<repo>

Download and Configure the Runner.

Once you create this directory and enter it, you will then download and configure the runner. The steps are reproduce below, but you should follow the instructions from the “add new self-hosted runner” page after clicking on “New self-hosted runner”.

Patch the Runner Folder for use as a User Systemd Service.

Apply this patch to modify the directory to use user systemd modules.

Enable linger for your user

Use this command to enable linger for your user.

This allows your user-level systemd services to run when you are not logged into the system and auto-start when the system is rebooted.

Note: Use loginctl disable-linger to remove linger and ls /var/lib/systemd/linger to view the users with linger set.

Use the svc.sh script to install and manage the runner service.

Install service

Start service and check status.

Note: The above install adds the service to auto start on reboot. If you want to disable or enable this auto starting of the service run.

or

To stop the service run

To uninstall the service run

Creating a secure, human-in-the-loop CI pipeline for public repos

2. Trigger on issue_comment: this is the event that triggers the CI pipeline. The types: [created] ensures that the pipeline is triggered only when a new comment is made and not when an existing comment is edited.

   NOTE: in GitHub Actions PRs are issues, so the issue_comment event is used to trigger the pipeline when a PR comment is made.

3. Verify Actor: and "actor" is any user writing a comment on the PR. This step verifies that the actor is an authorized user to trigger the CI pipeline. The following is an example of how to verify the actor in the workflow yaml file. ACTOR_TOKEN puts the current "actor" within the delimiter and checks if it is in the list of authorized users. If it is, it triggers the pipeline. If not, it skips all subsequent steps.
5. Create PR status: this step creates a status check on the PR extracting information from the json information generated in the previous step. This steps allows for seamless integration with the typical checks interface for a PR along with other CI workflow. The status check is created as a "pending" status and the URL is linked to the current pipeline run before the actual tests run.
6. Run tests: the following steps continue the pipeline tests and they are specific to each workflow reusing these steps.

7. Report PR status: this step reports the status of the pipeline to the PR. The status is updated to "success" if the tests pass and "failure" if the tests fail. The URL is linked to the current pipeline run to update the PR status created in step 4.

ExCL uses Modules to manage software environments efficiently. Modules allow users to load, unload, and switch between different software versions without modifying system paths manually. Please let us know if there is a software package you would like us to make available via a module.

Loading a Module

To load a specific software module:

Example:

This makes Python 3.9 available for use.

You can also leave off the version number to load the default version.

Example:

Listing Available Modules

To see all available modules:

Checking Loaded Modules

To view currently loaded modules:

Unloading a Module

To remove a specific module:

Example:

Switching Between Versions

To switch from one module version to another:

Example:

Resetting the Environment

To clear all loaded modules and reset to the default environment:

Git (code revision management system) is installed on all ExCL systems on which it makes sense. Git operates as expected, except for external access.

If you require access to external git resources, you need to do a little more.

HTTP or HTTPS access

For HTTP or HTTPS access, make sure you have the following environment variables (they should be set by default, but may not be if you have altered your environment)

The proxy server has access to the full Oak Ridge network (open research only).

Git SSH Access

ssh can be used to clone repositories on the login node. In order to clone repositories on the internal nodes, the ssh config needs to be changed to use the login node as a proxy jump. Here is an example ssh config with jump proxies to code.ornl.gov, bitbucket.org, and github.com.

To configure git to always use ssh for code.ornl.gov repositories, use the config command below.

The recommended approach to access code.ornl.gov is to use SSH. To do this, you need to generate an SSH key and add it to your GitLab account. The following steps will guide you through the process.

1. Generate an SSH key.

1. Add the SSH key to your GitLab account.

SSH Keys for Authentication

Using SSH keys is the preferred way to authenticate your user and to authenticate with private Git repositories. For security, it is recommended to use an SSH keys encrypted with a passphrase.

Why not passwords?

ExCL will block your account after 3 failed attempts. Automatic login tools, e.g. VS Code, can easily exceed this limit using a cached password and auto-reconnect. For git repos with two-factor authentication, an application token/password must be created, and this password must be stored externally and is more cumbersome to use.

How to get started?

1. Set up a key pair:

SSH-Agent and SSH Forwarding

SSH-Agents cache SSH keys with passphrases, allowing them to be reused during the session. This is not needed with keys without a passphrase, since they can be used without decrypting.

SSH Forwarding: SSH agents can forward SSH keys to a remote system, making the keys available there as well.

How to get started?

2. Add key to agent

   • ssh-add or ssh-add [file] for non-default filenames.

   • Note: If you're running a mac and want to add an SSH key that's not one of the standard names (~/.ssh/id_rsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ecdsa_sk, ~/.ssh/id_ed25519, ~/.ssh/id_ed25519_sk, and ~/.ssh/id_dsa) use ssh-add --apple-use-keychain [file].

   • Check loaded keys with ssh-add –l.

3. Setup SSH forwarding in SSH config.

   • Log in and verify key is still available.

Git Basics

Git - a version control system that records the changes to a file or files which allows you to return to a previous version

ORNL Git Resources

When we talk about Git, we say that a repository stores files. This term means that you have a folder that is currently being tracked by Git. It is common, although optional, to use one of the Git repository (repo) services (GitHub, GitLab, BitBucket, etc.). You could easily set up Git tracking on your local machine only, but one of the perks to using Git is that you can share your files with others and a team can edit files collaboratively. The ability to collaborate is one of the many reasons why hosted Git repos are so popular.

Repository - the Git data structure which contains files and folders, as well as how the files/folders have changed over time

Accessing GitLab

• Choose the Blank project tab, create a name for the project, and select the "Visibility Level" that you prefer. Then click Create project.

• Notice that GitLab has provided instructions to perform Git setup and initialization of your repository. We will follow those instructions.

• (Optional) Prior to cloning the repository, consider adding your SSH key to your GitLab profile so you are not prompted for credentials after every commit. To add your public SSH key to GitLab:

  • Click on your user image in the top-right of the GitLab window.

  • Select Settings.

  • On the left, click SSH keys.

  • Paste your public SSH key in the box, provide a title, and save by clicking Add key.

Local Machine Setup

• First, use the command line to see if Git is installed. (Windows users may check their list of currently installed programs.)

  • To install or update Git using your package manager:

    • CentOS, RedHat:
    • Debian, Ubuntu:
• Setup Git with your access credentials to GitLab with the following commands (use your ORNL email):

  • You can review the information that you entered during set-up: git config --global --list

• Now, navigate to the location where you'd like to place your repository. For example:
• Clone the repository. A new folder is created, and Git starts tracking. Consult the repository information from the GitLab new repository window.

  Clone - is the equivalent of making a local copy on your computer
• GitLab also recommends the creation of a README.md file to describe the repository. (We will edit the contents of the README.md file later.)
• The next three steps consist of adding, committing, and pushing from your local machine to GitLab.

  Add - includes the added files in the content that you want to save Commit - creates a "snapshot" of the repository at that moment and uses the changes from the "added" files Push - moves/uploads the local changes (or snapshot) to the remote GitLab repository

  • (Optional) If you like, you can refresh your browser page, and you can see that the README.md file is now in your repository.

Using Branches to Make Changes

Branches are created as a way to separate content that is still under development. One way to think about a branch is as a copy of the content of a repository at a point in time. You'll then make your changes on the copy before then integrating the changes back into the original. For example, if you were using your GitLab repo to host a website, you probably would not want incomplete content shown to those who would visit your site. Instead, you can create a branch, make edits to the files there, then merge your development branch back into the master branch, which is the default branch. Additionally, branches are commonly used when multiple individuals work out of a single repository.

Branch - a version of the repository that splits from the primary version Merge - using the changes from one branch and adding them to another

• A branch checkout enables you to make changes to files without changing the content of the master branch. To create and checkout a branch called "adding-readme":

Checkout - Git command to change branches

• Now we edit the README.md file to add a description of the repository. The file needs to be opened with a text editor (nano, vim, emacs, etc.).
• • To type in vi, press i for insert. Now you can add content.

  • To save your changes and exit vi, press <esc> to leave editing, then type :wq which writes (saves) and quits.

• As before, we need to add, commit, and push the changes to the GitLab repository.

  • In future pushes, you can simplify the last command by typing only git push. However, the first time you push to a new branch, you have to tell GitLab that you have created a new branch on your computer and the changes that you are pushing should be pushed to a new remote branch called adding-readme.

Merging Content from a Development Branch to the Master Branch

After completing the previous section, we have two branches: adding-readme and master. We are ready to move the adding-readme content to the master branch.

You can create a merge request using the GitLab GUI.

• From the left menu panel in Gitlab (when viewing the repository), select Merge Request then the green New merge request button.

• Select your branch on the "Source Branch" side (adding-readme).

  • Target branch is master.

  • Click Compare branches and continue.

• You can add as much information to the next screen as you like, but the only thing needed is:

  • Assign to: < Project Owner, etc. >

    • In our case, we are the project owner, so we may assign the merge request to ourselves.

  • Click Submit merge request.

• On the next page, click the green Merge button.

From the left menu panel in Gitlab, select Overview to see the new README.md content.

External Reference Material

Sometimes Git repository sites use different terminology, i.e., merge request vs. pull request. To reference the glossaries:

Ready to Learn More?