Snapshot installs as a single per-cluster Helm release. Install one release for the cluster: every node agent watches checkpoint and restore pods in all namespaces. Install it in its own namespace, and run GPU workloads in separate namespaces.
Review the Prerequisites before installing.
- The operator Deployment, which reconciles
SnapshotJob,PodSnapshot, andPodSnapshotContent. - The node agent DaemonSet on eligible GPU nodes. It runs privileged to perform checkpoint and restore.
- The snapshot volume claim,
snapshot-pvc, or the wiring to an existing claim. - Cluster-scoped RBAC for the agent and the operator.
- The seccomp profile that CRIU needs.
The node agent needs elevated privileges to checkpoint and restore processes:
- The
snapshot-agentDaemonSet runs privileged, withhostPID,hostIPC, andhostNetwork, so it can run CRIU andcuda-checkpointagainst live processes on the node. Workloads do not need to be privileged. Only the agent does. - The agent's namespace must allow privileged pods. On clusters that enforce
Pod Security Standards,
apply the
privilegedlevel, or an equivalent exception, to that namespace before you install. - CRIU needs a seccomp profile to checkpoint and restore. The chart installs the profile the agent needs.
The agent uses ClusterFirstWithHostNet DNS policy so its host-networked Pod
can resolve Kubernetes Services. This also applies to a remote metadata API
exposed through a Service on a Liqo provider.
Install the published chart. Replace <version> with the release you want from
the releases page. Every
command on this page uses the same placeholder.
helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
--version <version> \
--namespace snapshot --create-namespaceFor every value you can set, see Helm Chart Values.
- By default, the chart creates a
ReadWriteManysnapshot volume that every snapshot shares. - If your cluster has no default storage class, add
--set storage.pvc.storageClass=<storage-class>. - To reuse an existing
ReadWriteManyclaim, add--set storage.pvc.create=false --set storage.pvc.name=<pvc-name>. The chart does not check the claim's access mode, so verify that it isReadWriteManybefore you install.
See Configure Storage for the volume model and other options.
- Each agent pod runs two containers: the agent and the PageBroker sidecar. PageBroker stages snapshot data in memory and publishes it to the snapshot volume.
- By default, the agent pod requests 10 CPU and 33Gi of memory on each GPU node.
- The pod's memory limit sets the largest snapshot it can take.
See PageBroker Staging for how to size it.
OpenShift needs two settings that a standard install does not, for its container runtime and its security rules. Set these values when you install:
runtime.type=crio. OpenShift uses CRI-O as its container runtime.openshift.enabled=true. This creates the SCC-use RBAC and the required-SCC pod annotations for both components. The agent uses theprivilegedSCC, and the operator usesanyuid.
The chart creates these SCC permissions only while rbac.create is true, the
default. If you set it to false, grant the agent and the operator permission
to use their SCCs yourself.
Create the snapshot volume through a ReadWriteMany storage class, such as an
EFS storage class:
export RWX_STORAGE_CLASS=efs-rwx
helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
--version <version> \
--namespace snapshot --create-namespace \
--set storage.pvc.create=true \
--set storage.pvc.storageClass="${RWX_STORAGE_CLASS}" \
--set runtime.type=crio \
--set openshift.enabled=trueOr reuse an existing ReadWriteMany claim in the installation namespace:
export EXISTING_RWX_PVC=snapshot-efs-pvc
helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
--version <version> \
--namespace snapshot --create-namespace \
--set storage.pvc.create=false \
--set storage.pvc.name="${EXISTING_RWX_PVC}" \
--set runtime.type=crio \
--set openshift.enabled=trueRKE2 and k3s are Kubernetes distributions from Rancher. Both run containerd, but they keep its socket and its storage in their own directories, not in the standard locations. The node agent needs both paths to reach containerd, so set both values together:
| Value | RKE2 | k3s |
|---|---|---|
runtime.socketPath |
/run/k3s/containerd/containerd.sock |
/run/k3s/containerd/containerd.sock |
runtime.storageDir |
/var/lib/rancher/rke2/agent/containerd |
/var/lib/rancher/k3s/agent/containerd |
For example, on RKE2:
helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
--version <version> \
--namespace snapshot --create-namespace \
--set runtime.socketPath=/run/k3s/containerd/containerd.sock \
--set runtime.storageDir=/var/lib/rancher/rke2/agent/containerdSet runtime.storageDir to the exact host path. The chart mounts it into the
agent at the same path, and the agent reads the container's writable layer at
the path containerd records. It is not a relocatable prefix.
Mirror these images when your cluster pulls from a private or air-gapped registry:
- Operator:
ghcr.io/ai-dynamo/snapshot/operator - Node agent:
ghcr.io/ai-dynamo/snapshot/agent - PageBroker:
ghcr.io/ai-dynamo/snapshot/pagebroker, always pulled at the node agent's tag - busybox:
busybox:1.37.0, digest-pinned, from Docker Hub. A short-lived init container uses it to write the seccomp profile. Override it withdaemonset.initContainer.image.
- The node agent uses public GHCR images by default and needs no image pull secret.
- Set
daemonset.imagePullSecretsonly when you override the agent image with a private one. - When you upgrade, see Upgrade Snapshot and the CRDs.
kubectl get pods --namespace snapshot
kubectl get pvc --namespace snapshot
kubectl rollout status daemonset/snapshot-agent --namespace snapshotThe operator and the snapshot-agent DaemonSet become ready once the node agent
and its PageBroker sidecar are running on each GPU node. Each agent pod reports
2/2 containers ready. The snapshot volume claim reports Bound.
helm uninstall snapshot --namespace snapshotChart-created snapshot volumes are retained on uninstall, so snapshots survive removal. See Retention. To clear the stored snapshots, delete the claim yourself:
kubectl delete pvc <pvc-name> --namespace snapshot