Skip to content

Latest commit

 

History

History
389 lines (310 loc) · 18.4 KB

AGENTS.md

File metadata and controls

389 lines (310 loc) · 18.4 KB

AGENTS.md

Guidance for AI coding agents working in this repository. Human contributors should start with CONTRIBUTING.md; everything there applies here too.

Note: "agent" in this file means an AI coding assistant. The agent/ module is a different thing entirely — the node-level Snapshot agent daemon.

What this project is

Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads. It checkpoints a fully initialized GPU pod — running process, CPU and GPU memory — and restores that state on another compatible node. It provides primitives, not orchestration.

Agent skills

Repository agent skills live in .agents/skills/ and are versioned with the checkout. Use the snep skill when drafting or revising a Snapshot Enhancement Proposal (SNEP); it guides the proposal section by section and follows the process in docs/design/README.md.

Repository layout

Path Go module What lives there
api/ yes CRD types (PodSnapshot, PodSnapshotContent, SnapshotJob) and generated CRD YAML in api/v1alpha1/crds
agent/ yes Node-level daemon that drives CRIU and CUDA checkpointing. Linux only (amd64, arm64)
operator/ yes Cluster controller that reconciles the CRDs
charts/snapshot/ — Helm chart; charts/snapshot/crds is a generated copy of the CRDs
e2e/ — Python end-to-end suite (uv/pyproject), not Go
docs/ — User and developer documentation
hack/ — Build tooling, license boilerplate, base-package capture

Three separate Go modules — api, agent, and operator — each with its own go.mod and its own Makefile. The root Makefile fans out to all three.

Build and test

Run everything from the repository root. Go 1.27.3 — the version is pinned in hack/tools.mk, go.work, all three go.mod files, agent/Dockerfile, and the workflows, and they must agree.

make check        # default goal: the full local gate (see below)
make test         # unit tests across api, agent, operator
make build        # build agent and operator binaries
make lint         # golangci-lint across all three modules
make fmt          # format
make generate     # regenerate CRDs from api/ types

make check is the one that matters before opening a pull request. It runs, in order: verify-crds, install-tools, generate, pagebroker-check-generated, add-license-headers, fmt, tidy, verify-license-headers, lint, govulncheck, helm-lint, verify-toc, and verify-troubleshooting-steps. It deliberately does not run in parallel (.NOTPARALLEL:) because the stages mutate files that later stages read.

To scope work to one module, use its own Makefile — make -C operator test.

Building the agent from macOS

The agent is Linux-only (amd64 and arm64), so it does not build natively on a Mac. Use the containerized targets:

make linux-build   # builds agent/ inside a golang container
make linux-test    # runs agent/ tests inside a golang container

Both require Docker.

Conventions that will fail CI if you miss them

  1. Every commit must be signed off (DCO). Use git commit -s. Unsigned commits fail the DCO check and cannot be merged. See CONTRIBUTING.md.

  2. Every pull request must link an approved issue. The Validate Issue Reference check fails when a pull request references no issue, or only issues that are closed or not yet labeled approved. Open the issue first and wait for a maintainer to apply approved before investing significant work. Do not open a pull request that skips this step.

  3. SPDX license headers are required on new files. make add-license-headers adds them from hack/boilerplate.addlicense.txt; verify-license-headers checks them in CI. Generated CRDs are exempt.

  4. CRDs are generated, not hand-edited. Edit the types under api/v1alpha1, then run make generate. The chart copy in charts/snapshot/crds must stay in sync with api/v1alpha1/crds — verify-crds fails on drift and tells you to run make generate and commit the result.

  5. The agent base image is pinned in one place. AGENT_BASE_IMAGE is read out of agent/Dockerfile. If you change it, re-run make capture-base-packages so the committed package baseline matches; verify-base-packages enforces this.

Preferred and deprecated patterns

Concrete examples of the shape a change should take here. Each is enforced somewhere — a linter, a make target, or a CI job — so getting it wrong shows up as a failure rather than a review comment.

Read the control directory from the canonical environment variable. LegacySnapshotControlDirEnv exists only so workload images built against the old name keep working during the migration window. New code must not read it; operator/.golangci.yml scopes the deprecation exclusion to the single file allowed to reference it.

// Good
dir := os.Getenv(podcontract.SnapshotControlDirEnv)        // SNAPSHOT_CONTROL_DIR

// Bad — deprecated, kept only for backward compatibility
dir := os.Getenv(podcontract.LegacySnapshotControlDirEnv)  // DYN_SNAPSHOT_CONTROL_DIR

Log through the context logger, not stdout. Reconcilers get a logger carrying the request's identity; printing loses it. There is no fmt.Print in operator/internal or agent/internal — do not add the first one.

// Good
log.FromContext(ctx).Error(err, "Failed to list PodSnapshots", "pod", key)

// Bad — no request context, invisible to log scraping
fmt.Printf("failed to list PodSnapshots for %s: %v\n", key, err)

Generate CRDs; never hand-edit them. verify-crds diffs the two copies and fails on drift.

# Good — edit the Go types, then regenerate both copies
vim api/v1alpha1/podsnapshot_types.go && make generate

# Bad — the next `make generate` overwrites it, and CI fails first
vim charts/snapshot/crds/nvidia.com_podsnapshots.yaml

Pin base images by digest. A tag is mutable, so the image that ships is whatever the registry served that day. Docker resolves tag@digest by digest and never validates the tag against it, so the two must move together.

# Good
FROM ubuntu:24.04@sha256:224a1869083a311ef3f13648a154ba79832fbef6364d31493642ca03082da254

# Bad — mutable tag
FROM ubuntu:24.04

Move the Go toolchain version everywhere at once. Bumping one place leaves the shipped binary built by a toolchain CI never validated, so .github/dependabot.yml ignores major and minor updates to the golang image specifically to stop that happening automatically.

# Good — find every reference first, change them together
rg -l '1\.27\.1' go.work */go.mod hack/tools.mk agent/Dockerfile .github/workflows

# Bad — the operator now builds on a toolchain nothing else agrees with
vim operator/Dockerfile   # FROM golang:1.28-alpine

Document a feature's version floor on the Prerequisites page, not just in an error message. Snapshot's general minimum driver and GPU Operator version lives in the README's Prerequisites and in Baseline Prerequisites. A feature that needs more than that floor must add a row to docs/get-started/prerequisites.md in the same change — see CONTRIBUTING.md. A version requirement that only exists as a runtime failure string is invisible to anyone deciding whether to adopt the feature.

// Good — floor is documented in docs/get-started/prerequisites.md and enforced here
if !driverSupports(minLaunchJobDriverVersion) {
    return fmt.Errorf("cuda-checkpoint --launch-job requires NVIDIA driver %s or newer", minLaunchJobDriverVersion)
}

// Bad — the 610 floor exists only in this string; nothing else references it
echo "cuda-checkpoint --launch-job requires NVIDIA driver 610 or newer" >&2

Secrets and credentials

Never commit credentials, secrets, API keys, tokens, kubeconfigs, or .env files. This repository contains none, and no change should introduce any. Specifically:

  • Do not add a .env file, a kubeconfig, a cloud credential file, or a private key to the tree, and do not paste any of them into code, tests, fixtures, comments, commit messages, or pull request descriptions.
  • Do not hardcode a registry password, GitHub token, or NGC/NVIDIA API key. Workflows read credentials from GitHub Actions secrets (${{ secrets.* }}); local runs read them from your own environment.
  • Do not echo secrets in workflow run: blocks or test output — a value printed in a public CI log is disclosed.
  • Test fixtures use obviously fake placeholders. Never copy a real token into one, even an expired one.
  • If you find a committed secret, do not fix it in a public pull request. Treat it as a vulnerability and follow SECURITY.md; the credential needs rotating, and a public commit reverting it advertises the leak.

When a task genuinely needs a credential, read it from an environment variable and document the variable name — never the value.

Documentation and names

Before you add or change Markdown under docs/, a README, or CHANGELOG.md, or add a version requirement such as a minimum driver or CUDA version, read docs/development/writing-docs.md. Before you introduce a name (a field, Helm value, annotation, label, environment variable, event reason, flag, file, or a new term in prose), read docs/development/naming.md. A name that a user sees describes the capability, not the mechanism behind it.

Working style

  • Match the surrounding code. The repository favors explanatory comments on non-obvious decisions — see the .NOTPARALLEL: and AGENT_BASE_IMAGE comments in the root Makefile for the register to aim for.
  • Add or update tests alongside behavior changes. Controller logic under agent/internal and operator/internal has substantial table-driven test coverage; follow the existing patterns rather than inventing new harnesses.
  • The operator's Pod informer cache is trimmed. TrimPodForCache in operator/internal/controller/pod_cache.go keeps metadata (minus managedFields), spec.nodeName, status.phase and the container and init-container statuses, and clears the rest of spec and status. A controller that reads any other Pod field from the cache gets a zero value, so extend the transform and its test in the same change.
  • Do not edit generated files (zz_generated*.go, both CRD directories).
  • Do not bump dependencies as a side effect of an unrelated change; Dependabot handles routine updates.
  • Never put a security vulnerability in a public issue or pull request — follow SECURITY.md.

Where to look first


doublewordai/snapshot fork layout

Integration fork of ai-dynamo/snapshot. The fleet's Snapshot operator and node agent images are built from a commit of this repo's main.

Branches

  • upstream-base: exactly the upstream revision the fork is based on. Currently 77010731725572ace26c13c9f4a98652debcc115 (2026-11-10 upstream main). Never carries our commits.
  • main: upstream-base plus every patch branch below, merged with --no-ff in stack order. git log --merges upstream-base..main is the patch list.
  • upstream-pr/<topic>: a change we intend to land upstream, based on upstream-base.
  • vendor/<topic>: a Doubleword-only change, based on upstream-base.
  • archive/* and other branches are history and are not part of any image. Old vendor branches not listed in the current stack are inactive.

Rules

  • No backports. Do not cherry-pick upstream commits onto main; a fix in a newer release arrives by moving upstream-base.
  • One branch per patch, atomic, with the reason in the commit message.
  • Moving to a new base: save backup refs, point upstream-base at the reviewed revision, rebase each patch still needed, drop patches the new base contains, rebuild main with merges, and update the stack list.
  • Prepare changes locally. Publication and image builds are separate human operations; do not push or trigger remote jobs from an agent session.

Current stack

  • vendor/fork-layout: this section.
  • vendor/helm-unittest: pin v1.0.3 for the Helm 3 test toolchain.
  • vendor/fork-publishing: skip the upstream artifact publisher and NVIDIA GPU deployment workflow in this fork; ordinary CPU CI remains enabled. Doubleword releases use the separate doublewordai/dynamo-images pipeline.
  • vendor/urllib3: preserve the fork's merged urllib3 2.8.0 lockfile update while upstream still pins 2.7.0.
  • vendor/mig-staging-safety: refuse a CUDA operation when the MIG probe fails, and validate every supported staging quantity form before applying the cap.
  • vendor/liqo-control-plane: preserve the consumer metadata plane, provider-local runtime, provenance admission, remote cleanup delegation, projected credentials and webhook startup fixes. Remote agents still require legacy content without spec.storage; artifact-addressed content is refused.
  • vendor/agent-cluster-dns: use ClusterFirstWithHostNet for the node agent.
  • vendor/certifi-e2e: preserve the merged certifi 2026.10.19 lockfile update.
  • vendor/certifi-benchmarks: preserve the merged certifi 2026.10.18 update.

The previous MIG refusal and staging-cap backports are now in upstream and are not replayed. Their fork-specific safety follow-ups remain in the stack.

The old vendor/cuinterpose patch is no longer carried: upstream includes its protocol, preload frontend, VMM/IPC tracking, multicast, coordinator, launcher and opt-in agent/operator integration, plus newer peer-allocation fixes. Use upstream's implementation rather than replaying the old stack. cuInterpose remains opt-in via nvidia.com/snapshot-multi-gpu: "enabled" on capture Pod templates. Single-node multi-GPU qualification is still required for each target model, engine, GPU topology and driver; an upstream merge or CPU test pass does not establish GLM-5.3 or Kimi-K3 restore support. Multi-node workloads remain outside Snapshot's current supported scope.

Dynamo's Snapshot API replacement and Snapshot image source pin must move together with this fork. PageBroker remains mandatory. This upstream base adds S3 storage, application lifecycle hooks, checkpoint readiness probes, dump retries and parallel rank restore. It also renames user-facing capture fields to checkpoint fields. Downstream integrations must preserve the selected store and published artifact identities and use the matching API. Qualify the matching backend before deployment. Keep operator.bindNewContents disabled for the Liqo metadata plane, which still uses provider-local legacy content and cleanup.

The PID-1 restore proxy requires NRI v0.10 or newer (containerd 2.2 or newer). On older runtimes or nodes without NRI, the agent now restores without the proxy. Destination containers must run an inert command and probe the actual workload. Without the proxy, the container does not follow workload exit and the workload does not receive the container's termination signals. See docs/reference/restore-pod-contract.md#nodes-without-the-restore-proxy and qualify lifecycle and drain behavior before choosing that fallback.

upstream-pr/runtime-storage-path remains dropped: upstream provides runtime.storageDir; migrate old runtime.storagePath values.

Release and upgrade requirements

Pushing this fork updates source and runs CI. It does not publish Doubleword images: .github/workflows/push-artifacts.yaml is gated to ai-dynamo/snapshot, whose registry and cache it writes. All jobs that publish images, charts or caches depend on that gate.

Doubleword's publisher is doublewordai/dynamo-images, workflow .github/workflows/build-images.yml. After the Snapshot source is published, update that repository's immutable Snapshot source pin and the Dynamo Snapshot API dependency together. The builder includes Snapshot operator, agent and PageBroker targets, and scripts/point-snapshot-chart.sh rewrites all three image repositories. Agent and PageBroker must come from the same Snapshot revision and use the matching image tag expected by the chart. A green source CI run does not establish a complete Doubleword release.

For workloads migrating from the old vendor cuInterpose stack:

  • Replace nvidia.com/cuinterpose-enabled: "true" on capture Pod templates with nvidia.com/snapshot-multi-gpu: "enabled". The intermediate nvidia.com/cuda-shared-memory-support key is also no longer read; the new key accepts enabled or disabled, not booleans.
  • Create fresh captures with the new agent/operator bundle. Old cuInterpose manifests use frontendSHA256 and coreSHA256; this base requires the libraries map and an explicit pids list, so it rejects those old captures. Do not rewrite existing manifests to bypass validation. New captures have new snapshot/content identities and must replace the selected restore source.
  • Keep old artifacts and their matching software until rollback and active consumers have been checked. Test a restore from the new capture before switching production workloads to it.
  • Install the compatible agent, PageBroker, operator and CRDs together. Verify NRI v0.10+ for the PID-1 proxy, or explicitly qualify the documented proxy-free lifecycle with an inert destination command and workload probe.