DVC

Getting started with DVC on ExCL systems.

DVC (Data Version Control) is a content-addressable data management system that integrates with Git. On ExCL, DVC enables shared, deduplicated storage across projects using a common cache.

DVC works well with ExCL. On ExCL, data can be stored in a shared cache in the project folder. Outside of ExCL, this same cache, can be used as the remote data storage.

---

🚀 What You Get on ExCL with shared data cache

• Shared cache across repositories → no duplicate datasets

• Content-addressed storage → automatic change detection

• Reproducible pipelines (optional)

• Git-based versioning for data pointers

• Flexible remote syncing (for off-cluster use)

---

⚡ Quick Start (New Repository)

1. Install DVC

uv tool install dvc[ssh,s3]

---

2. Initialize a Repository

git init
dvc init

# Auto-stage DVC changes into git
dvc config core.autostage true

---

3. Configure Shared Cache (ExCL)

# Set cache directory path to shared project cache location
dvc cache dir /auto/project/<project>/dvc-cache

# Enable group sharing of cache
dvc config cache.shared group

# Use symlinks (recommended on ExCL)
dvc config cache.type symlink

This can also be skipped and done instead at the instance level to better support using the repo on non-ExCL systems. See Post Clone Steps on ExCL | ExCL User Docs.

---

📦 Adding Data

dvc add data/my-dataset

git add data/my-dataset.dvc .gitignore # Not needed if autostage is true.
git commit -m "Track dataset with DVC"

dvc push
git push

---

🔄 Getting Data

dvc checkout

If cache settings changed:

dvc checkout --relink

---

🧰 Helpful Commands

# Show configuration (with source)
dvc config -l --show-origin

# Check data status vs cache/remote
dvc status -c

# Fix workspace links after config change
dvc checkout --relink

# Make git repo group-writable (shared repos)
# This is useful if you want to share a git DVC repo in project space.
git config core.sharedRepository group

---

🧑‍🤝‍🧑 Cloning a DVC Repository and Post Clone Steps

On ExCL (shared cache)

git clone <repo>
cd <repo>

# Set cache directory path to shared location
dvc config --local cache.dir /auto/project/<project>/dvc-cache
# Share cache with your group
dvc config --local cache.shared group
# Use symlinks to map files from the cache.
dvc config --local cache.type symlink

dvc checkout

With ExCL’s shared cache, dvc push and dvc pull are not needed, only dvc checkout and dvc checkout --relink.

---

Outside ExCL (remote usage)

git clone <repo>
cd <repo>

# Set the remote to be ExCL's DVC cache
dvc remote add -d --local excl ssh://login.excl.ornl.gov/auto/project/<project>/dvc-cache

# Default is reflink,copy so if reflink is not supported, you will want to change it. Check for supported types with `dvc version`. Symlink is recommended if reflink is not an option.
dvc config --local cache.type symlink

# Sync data
dvc pull

For systems outside of ExCL’s, treat ExCL as a DVC remote and use dvc push and dvc pull to synchronize files to and from your local cache.

---

🧠 Mental Model

• Git tracks small metadata files (.dvc)

• DVC cache stores actual data (by hash)

• Workspace files are symlinks into the cache

• Multiple repos → same shared cache → no duplication

---

🗒️ Best Practices on ExCL

• Use a shared project cache (/auto/project/…)

• Always run dvc checkout after cloning

• Prefer cache.type symlink for data sharing and clarity

• Avoid unnecessary pipeline re-runs (be careful with dependencies)

• Periodically review cache usage (dvc gc with care)

---

📊 When to Use DVC vs. <Insert Existing Workflow>

Use DVC when:

• You want reproducibility and versioned datasets

• Multiple repos need to share large datasets

• You want content-based tracking instead of paths

Use existing workflow when:

• You have large exploratory runs (e.g., DEFFE)

• Data is expensive to recompute, or you have bespoke processes.

• You prefer manual control and simpler tooling

---

🔁 Key Change from Simple Data Storage Approach

Instead of:

/path/to/data/file

You reference:

(repo, git commit, dvc-tracked file)

→ Data becomes portable, versioned, and reproducible

---

🧹 Notes on Cache Management

• Cache grows over time (content-addressed)

• Use:

dvc gc

Only run with care in shared environments (can remove needed data)

Note that enabling soft/hard links causes DVC to protect the linked data because editing them in-place would corrupt the cache. See dvc unprotect.

Using dvc gc with a shared cache may delete data needed in another project! See more info about cleaning a shared cache safely.