Guidance for AI coding agents working in this repository. Human contributors should start with CONTRIBUTING.md; everything there applies here too.
Note: "agent" in this file means an AI coding assistant. The
agent/module is a different thing entirely — the node-level Snapshot agent daemon.
Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads. It checkpoints a fully initialized GPU pod — running process, CPU and GPU memory — and restores that state on another compatible node. It provides primitives, not orchestration.
Repository agent skills live in .agents/skills/ and are versioned with the
checkout. Use the snep skill when drafting or revising a Snapshot Enhancement
Proposal (SNEP); it guides the proposal section by section and follows the
process in docs/design/README.md.
| Path | Go module | What lives there |
|---|---|---|
api/ |
yes | CRD types (PodSnapshot, PodSnapshotContent, SnapshotJob) and generated CRD YAML in api/v1alpha1/crds |
agent/ |
yes | Node-level daemon that drives CRIU and CUDA checkpointing. Linux only (amd64, arm64) |
operator/ |
yes | Cluster controller that reconciles the CRDs |
charts/snapshot/ |
— | Helm chart; charts/snapshot/crds is a generated copy of the CRDs |
e2e/ |
— | Python end-to-end suite (uv/pyproject), not Go |
docs/ |
— | User and developer documentation |
hack/ |
— | Build tooling, license boilerplate, base-package capture |
Three separate Go modules — api, agent, and operator — each with its own
go.mod and its own Makefile. The root Makefile fans out to all three.
Run everything from the repository root. Go 1.27.3 — the version is pinned in
hack/tools.mk, go.work, all three go.mod files, agent/Dockerfile, and
the workflows, and they must agree.
make check # default goal: the full local gate (see below)
make test # unit tests across api, agent, operator
make build # build agent and operator binaries
make lint # golangci-lint across all three modules
make fmt # format
make generate # regenerate CRDs from api/ typesmake check is the one that matters before opening a pull request. It runs, in
order: verify-crds, install-tools, generate, pagebroker-check-generated,
add-license-headers, fmt, tidy, verify-license-headers, lint,
govulncheck, helm-lint, verify-toc, and verify-troubleshooting-steps.
It deliberately does not run in parallel
(.NOTPARALLEL:) because the stages mutate files that later stages read.
To scope work to one module, use its own Makefile — make -C operator test.
The agent is Linux-only (amd64 and arm64), so it does not build natively on a Mac. Use the containerized targets:
make linux-build # builds agent/ inside a golang container
make linux-test # runs agent/ tests inside a golang containerBoth require Docker.
-
Every commit must be signed off (DCO). Use
git commit -s. Unsigned commits fail the DCO check and cannot be merged. See CONTRIBUTING.md. -
Every pull request must link an approved issue. The
Validate Issue Referencecheck fails when a pull request references no issue, or only issues that are closed or not yet labeledapproved. Open the issue first and wait for a maintainer to applyapprovedbefore investing significant work. Do not open a pull request that skips this step. -
SPDX license headers are required on new files.
make add-license-headersadds them fromhack/boilerplate.addlicense.txt;verify-license-headerschecks them in CI. Generated CRDs are exempt. -
CRDs are generated, not hand-edited. Edit the types under
api/v1alpha1, then runmake generate. The chart copy incharts/snapshot/crdsmust stay in sync withapi/v1alpha1/crds—verify-crdsfails on drift and tells you to runmake generateand commit the result. -
The agent base image is pinned in one place.
AGENT_BASE_IMAGEis read out ofagent/Dockerfile. If you change it, re-runmake capture-base-packagesso the committed package baseline matches;verify-base-packagesenforces this.
Concrete examples of the shape a change should take here. Each is enforced
somewhere — a linter, a make target, or a CI job — so getting it wrong shows
up as a failure rather than a review comment.
Read the control directory from the canonical environment variable.
LegacySnapshotControlDirEnv exists only so workload images built against the
old name keep working during the migration window. New code must not read it;
operator/.golangci.yml scopes the deprecation exclusion to the single file
allowed to reference it.
// Good
dir := os.Getenv(podcontract.SnapshotControlDirEnv) // SNAPSHOT_CONTROL_DIR
// Bad — deprecated, kept only for backward compatibility
dir := os.Getenv(podcontract.LegacySnapshotControlDirEnv) // DYN_SNAPSHOT_CONTROL_DIRLog through the context logger, not stdout. Reconcilers get a logger
carrying the request's identity; printing loses it. There is no fmt.Print in
operator/internal or agent/internal — do not add the first one.
// Good
log.FromContext(ctx).Error(err, "Failed to list PodSnapshots", "pod", key)
// Bad — no request context, invisible to log scraping
fmt.Printf("failed to list PodSnapshots for %s: %v\n", key, err)Generate CRDs; never hand-edit them. verify-crds diffs the two copies and
fails on drift.
# Good — edit the Go types, then regenerate both copies
vim api/v1alpha1/podsnapshot_types.go && make generate
# Bad — the next `make generate` overwrites it, and CI fails first
vim charts/snapshot/crds/nvidia.com_podsnapshots.yamlPin base images by digest. A tag is mutable, so the image that ships is
whatever the registry served that day. Docker resolves tag@digest by digest
and never validates the tag against it, so the two must move together.
# Good
FROM ubuntu:24.04@sha256:224a1869083a311ef3f13648a154ba79832fbef6364d31493642ca03082da254
# Bad — mutable tag
FROM ubuntu:24.04Move the Go toolchain version everywhere at once. Bumping one place leaves
the shipped binary built by a toolchain CI never validated, so
.github/dependabot.yml ignores major and minor updates to the golang image
specifically to stop that happening automatically.
# Good — find every reference first, change them together
rg -l '1\.27\.1' go.work */go.mod hack/tools.mk agent/Dockerfile .github/workflows
# Bad — the operator now builds on a toolchain nothing else agrees with
vim operator/Dockerfile # FROM golang:1.28-alpineDocument a feature's version floor on the Prerequisites page, not just in an error message. Snapshot's general minimum driver and GPU Operator version lives in the README's Prerequisites and in Baseline Prerequisites. A feature that needs more than that floor must add a row to docs/get-started/prerequisites.md in the same change — see CONTRIBUTING.md. A version requirement that only exists as a runtime failure string is invisible to anyone deciding whether to adopt the feature.
// Good — floor is documented in docs/get-started/prerequisites.md and enforced here
if !driverSupports(minLaunchJobDriverVersion) {
return fmt.Errorf("cuda-checkpoint --launch-job requires NVIDIA driver %s or newer", minLaunchJobDriverVersion)
}
// Bad — the 610 floor exists only in this string; nothing else references it
echo "cuda-checkpoint --launch-job requires NVIDIA driver 610 or newer" >&2Never commit credentials, secrets, API keys, tokens, kubeconfigs, or .env
files. This repository contains none, and no change should introduce any.
Specifically:
- Do not add a
.envfile, a kubeconfig, a cloud credential file, or a private key to the tree, and do not paste any of them into code, tests, fixtures, comments, commit messages, or pull request descriptions. - Do not hardcode a registry password, GitHub token, or NGC/NVIDIA API key.
Workflows read credentials from GitHub Actions secrets
(
${{ secrets.* }}); local runs read them from your own environment. - Do not echo secrets in workflow
run:blocks or test output — a value printed in a public CI log is disclosed. - Test fixtures use obviously fake placeholders. Never copy a real token into one, even an expired one.
- If you find a committed secret, do not fix it in a public pull request. Treat it as a vulnerability and follow SECURITY.md; the credential needs rotating, and a public commit reverting it advertises the leak.
When a task genuinely needs a credential, read it from an environment variable and document the variable name — never the value.
Before you add or change Markdown under docs/, a README, or CHANGELOG.md, or
add a version requirement such as a minimum driver or CUDA version, read
docs/development/writing-docs.md.
Before you introduce a name (a field, Helm value, annotation, label,
environment variable, event reason, flag, file, or a new term in prose), read
docs/development/naming.md. A name that a user
sees describes the capability, not the mechanism behind it.
- Match the surrounding code. The repository favors explanatory comments on
non-obvious decisions — see the
.NOTPARALLEL:andAGENT_BASE_IMAGEcomments in the rootMakefilefor the register to aim for. - Add or update tests alongside behavior changes. Controller logic under
agent/internalandoperator/internalhas substantial table-driven test coverage; follow the existing patterns rather than inventing new harnesses. - The operator's Pod informer cache is trimmed.
TrimPodForCacheinoperator/internal/controller/pod_cache.gokeeps metadata (minusmanagedFields),spec.nodeName,status.phaseand the container and init-container statuses, and clears the rest of spec and status. A controller that reads any other Pod field from the cache gets a zero value, so extend the transform and its test in the same change. - Do not edit generated files (
zz_generated*.go, both CRD directories). - Do not bump dependencies as a side effect of an unrelated change; Dependabot handles routine updates.
- Never put a security vulnerability in a public issue or pull request — follow SECURITY.md.
- README.md — what the system does and how it is used
- docs/development/build-from-source.md — full build instructions
- docs/reference — API reference
- docs/operate — install, storage, and operational guides
- e2e/README.md — running the end-to-end suite
Integration fork of ai-dynamo/snapshot. The fleet's Snapshot operator and
node agent images are built from a commit of this repo's main.
upstream-base: exactly the upstream revision the fork is based on. Currently77010731725572ace26c13c9f4a98652debcc115(2026-11-10 upstreammain). Never carries our commits.main:upstream-baseplus every patch branch below, merged with--no-ffin stack order.git log --merges upstream-base..mainis the patch list.upstream-pr/<topic>: a change we intend to land upstream, based onupstream-base.vendor/<topic>: a Doubleword-only change, based onupstream-base.archive/*and other branches are history and are not part of any image. Old vendor branches not listed in the current stack are inactive.
- No backports. Do not cherry-pick upstream commits onto
main; a fix in a newer release arrives by movingupstream-base. - One branch per patch, atomic, with the reason in the commit message.
- Moving to a new base: save backup refs, point
upstream-baseat the reviewed revision, rebase each patch still needed, drop patches the new base contains, rebuildmainwith merges, and update the stack list. - Prepare changes locally. Publication and image builds are separate human operations; do not push or trigger remote jobs from an agent session.
vendor/fork-layout: this section.vendor/helm-unittest: pin v1.0.3 for the Helm 3 test toolchain.vendor/fork-publishing: skip the upstream artifact publisher and NVIDIA GPU deployment workflow in this fork; ordinary CPU CI remains enabled. Doubleword releases use the separatedoublewordai/dynamo-imagespipeline.vendor/urllib3: preserve the fork's merged urllib3 2.8.0 lockfile update while upstream still pins 2.7.0.vendor/mig-staging-safety: refuse a CUDA operation when the MIG probe fails, and validate every supported staging quantity form before applying the cap.vendor/liqo-control-plane: preserve the consumer metadata plane, provider-local runtime, provenance admission, remote cleanup delegation, projected credentials and webhook startup fixes. Remote agents still require legacy content withoutspec.storage; artifact-addressed content is refused.vendor/agent-cluster-dns: useClusterFirstWithHostNetfor the node agent.vendor/certifi-e2e: preserve the merged certifi 2026.10.19 lockfile update.vendor/certifi-benchmarks: preserve the merged certifi 2026.10.18 update.
The previous MIG refusal and staging-cap backports are now in upstream and are not replayed. Their fork-specific safety follow-ups remain in the stack.
The old vendor/cuinterpose patch is no longer carried: upstream includes
its protocol, preload frontend, VMM/IPC tracking, multicast, coordinator,
launcher and opt-in agent/operator integration, plus newer peer-allocation
fixes. Use upstream's implementation rather than replaying the old stack.
cuInterpose remains opt-in via nvidia.com/snapshot-multi-gpu: "enabled" on
capture Pod templates. Single-node multi-GPU qualification is still required
for each target model, engine, GPU topology and driver; an upstream merge or
CPU test pass does not establish GLM-5.3 or Kimi-K3 restore support. Multi-node
workloads remain outside Snapshot's current supported scope.
Dynamo's Snapshot API replacement and Snapshot image source pin must move
together with this fork. PageBroker remains mandatory. This upstream base
adds S3 storage, application lifecycle hooks, checkpoint readiness probes,
dump retries and parallel rank restore. It also renames user-facing capture
fields to checkpoint fields. Downstream integrations must preserve the
selected store and published artifact identities and use the matching API.
Qualify the matching backend before deployment. Keep operator.bindNewContents
disabled for the Liqo metadata plane, which still uses provider-local legacy
content and cleanup.
The PID-1 restore proxy requires NRI v0.10 or newer (containerd 2.2 or newer).
On older runtimes or nodes without NRI, the agent now restores without the
proxy. Destination containers must run an inert command and probe the actual
workload. Without the proxy, the container does not follow workload exit and
the workload does not receive the container's termination signals. See
docs/reference/restore-pod-contract.md#nodes-without-the-restore-proxy and
qualify lifecycle and drain behavior before choosing that fallback.
upstream-pr/runtime-storage-path remains dropped: upstream provides
runtime.storageDir; migrate old runtime.storagePath values.
Pushing this fork updates source and runs CI. It does not publish Doubleword
images: .github/workflows/push-artifacts.yaml is gated to
ai-dynamo/snapshot, whose registry and cache it writes. All jobs that publish
images, charts or caches depend on that gate.
Doubleword's publisher is doublewordai/dynamo-images, workflow
.github/workflows/build-images.yml. After the Snapshot source is published,
update that repository's immutable Snapshot source pin and the Dynamo
Snapshot API dependency together. The builder includes
Snapshot operator, agent and PageBroker targets, and
scripts/point-snapshot-chart.sh rewrites all three image repositories.
Agent and PageBroker must come from the same Snapshot revision and use the
matching image tag expected by the chart. A green source CI run does not
establish a complete Doubleword release.
For workloads migrating from the old vendor cuInterpose stack:
- Replace
nvidia.com/cuinterpose-enabled: "true"on capture Pod templates withnvidia.com/snapshot-multi-gpu: "enabled". The intermediatenvidia.com/cuda-shared-memory-supportkey is also no longer read; the new key acceptsenabledordisabled, not booleans. - Create fresh captures with the new agent/operator bundle. Old cuInterpose
manifests use
frontendSHA256andcoreSHA256; this base requires thelibrariesmap and an explicitpidslist, so it rejects those old captures. Do not rewrite existing manifests to bypass validation. New captures have new snapshot/content identities and must replace the selected restore source. - Keep old artifacts and their matching software until rollback and active consumers have been checked. Test a restore from the new capture before switching production workloads to it.
- Install the compatible agent, PageBroker, operator and CRDs together. Verify NRI v0.10+ for the PID-1 proxy, or explicitly qualify the documented proxy-free lifecycle with an inert destination command and workload probe.