A Pod requests restore by naming a ready PodSnapshot in its namespace. The
Snapshot node agent restores only Pods that implement this contract.
Go integrations should use:
restored, err := podcontract.Build(
pod,
podcontract.Request{
SnapshotName: snapshotName,
SourceContainer: checkpointedContainer,
Mappings: mappings,
},
podcontract.Options{
SeccompProfile: podcontract.DefaultSeccompLocalhostProfile,
EnableStartupGate: true,
},
)This release intentionally moves the Go Pod-contract API from
api/restorepod and api/v1alpha1 to api/podcontract without compatibility
aliases. Consumers must update restore annotations, mappings, status reasons,
RestoreOutcome, and ClassifyRestoreOutcome references to the new package.
Custom-resource types remain in api/v1alpha1.
podcontract.Build is a pure, atomic transformation: it returns a deep copy or
an error and never mutates its input. Reapplying it with the same arguments is
idempotent. It emits one canonical representation and validates that exact
output. podcontract.Validate checks only Snapshot's minimum restore protocol,
without mutation or Kubernetes API reads. Optional workload policy such as
startup gating and seccomp selection remains producer-owned. Conflicting
annotations, volumes, mounts, environment, and requested security settings are
rejected instead of overwritten.
Consumers can interpret the agent-owned nvidia.com/Restored Pod condition
without importing Snapshot's custom-resource API:
outcome := podcontract.ClassifyRestoreOutcome(pod.Status.Conditions)The package also owns the shared control-volume, environment, checkpoint-sentinel, and CUDA job-file names used by Snapshot-managed source and restore Pods.
The producer supplies SourceContainer from the referenced PodSnapshot.
Empty mappings restore that source into the same-named destination only. The
builder validates one-source-to-many-destination consistency but deliberately
performs no API read to discover the checkpointed source.
Non-Go producers can implement the declarative form directly. This fan-out
example restores the checkpointed main process into two destination containers:
apiVersion: v1
kind: Pod
metadata:
name: restored-worker
annotations:
nvidia.com/restore-from: worker-snapshot
nvidia.com/restore-container-map: main=engine-0,main=engine-1
spec:
volumes:
- name: snapshot-control
emptyDir: {}
containers:
- name: engine-0
image: worker:latest
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/block-iouring.json
# The source's command. Snapshot replaces it with its restore proxy.
command: ["/app/worker"]
env:
- name: SNAPSHOT_CONTROL_DIR
value: /snapshot-control
volumeMounts:
- name: snapshot-control
mountPath: /snapshot-control
subPath: engine-0
startupProbe:
exec:
command: ["cat", "/snapshot-control/restore-complete"]
timeoutSeconds: 1
periodSeconds: 1
failureThreshold: 1800
successThreshold: 1
- name: engine-1
image: worker:latest
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/block-iouring.json
command: ["/app/worker"]
env:
- name: SNAPSHOT_CONTROL_DIR
value: /snapshot-control
volumeMounts:
- name: snapshot-control
mountPath: /snapshot-control
subPath: engine-1
startupProbe:
exec:
command: ["cat", "/snapshot-control/restore-complete"]
timeoutSeconds: 1
periodSeconds: 1
failureThreshold: 1800
successThreshold: 1The container mapping is optional for a single same-name restore. Every
destination mounts the one shared snapshot-control emptyDir at
/snapshot-control with subPath equal to its container name. The canonical
SNAPSHOT_CONTROL_DIR environment variable points to that mount. It is a
required part of the minimum contract so every Snapshot-managed workload and
tool can discover the control directory without hard-coding its location. The
builder also injects the deprecated DYN_SNAPSHOT_CONTROL_DIR alias during
the migration window; hand-authored Pods may omit the alias.
The restore-complete startup probe shown above is optional workload lifecycle
policy. Set Options.EnableStartupGate when Kubernetes must withhold readiness
and liveness until restore completes. Kubernetes supports only one
startup probe, so the builder then replaces any existing startup probe with the
canonical restore-completion gate while preserving workload liveness and
readiness probes. Its failure threshold allows 1,800 consecutive one-second
failures; if restoration exceeds that budget, kubelet kills the container, and
the restore is lost (see restart policy below). The node agent always writes
the sentinel but does not require or inspect the optional probe.
Options.SeccompProfile controls the localhost profile on restore destination
containers only. The standard Snapshot Helm installation deploys
DefaultSeccompLocalhostProfile to block io_uring for CRIU. Applying it at
container scope preserves the Pod-level policy inherited by unrelated
sidecars. An empty value leaves seccomp unmanaged for environments that provide
the restriction elsewhere. A destination container must not override a
requested profile with a conflicting profile.
Snapshot replaces the command of every destination container with its restore
proxy, /snapshot-control/snapshot-restore-proxy. The Pod object is not
changed: the node agent's NRI plugin sets the process arguments when the
container is created. That needs NRI v0.10 or newer, enabled in the container
runtime: containerd 2.2 or newer. The proxy is the container's PID 1. It waits
for the restore, then follows the restored process: it forwards signals to it,
and exits with its exit code, so the container ends when the restored workload
does. If the proxy is not PID 1, the agent fails the restore. Keep the source's
command in restore Pods; it never runs. Other containers in the Pod, such as
sidecars, are left alone. Nodes whose runtime cannot run the proxy restore
without it; see Nodes Without the Restore Proxy.
The plugin decides which containers get the proxy when each container is
created. Without restore-container-map, it reads the PodSnapshot that
restore-from names, for the checkpointed container's name. Create the restore Pod
after that PodSnapshot exists; it does not have to be Ready yet. Until it
exists, container creation fails, and kubelet retries with a backoff of up to 5
minutes.
Probes belong to the workload. Kubernetes runs liveness and readiness probes
from container start, while the restore is still running and nothing in the
container answers them yet. A liveness probe that fails during that time kills
the container and loses the restore. Either set Options.EnableStartupGate,
which holds liveness and readiness back until restore-complete exists, or
give liveness probes an initialDelaySeconds or failureThreshold that covers
the longest restore you expect.
Restore Pods are expected to use restartPolicy: Never. A Pod gets one restore:
a container that starts again in the same Pod finds that restore already used,
and its proxy exits with code 112. With restartPolicy: Always or OnFailure,
as in a Deployment, the container then ends in CrashLoopBackOff once the
restored workload exits. Delete the Pod to restore again.
The proxy reserves two exit codes:
- 112 (
RestoreNotRepeatableExitCode): the Pod's one restore is used up. The restore failed, or an earlier container in the Pod already used it. - 113 (
RestoreProxyErrorExitCode): the proxy could not follow the restored process.
Every other exit code is the restored workload's own, or 128 plus the number of the signal that ended it.
On a node whose container runtime has no NRI, or NRI older than v0.10 (containerd older than 2.2, or NRI disabled), the agent restores without the proxy. Its log says so when it starts. The container's own command stays PID 1, and the agent restores the workload beside it. On such nodes:
- Give every destination container an inert, long-running command, such as
["sleep", "infinity"]. The source's command would start the workload from scratch next to the restored one. An inert command also works on nodes with the proxy, which replaces it. - Add a liveness probe that checks the workload itself, such as its API.
Without the proxy:
- The container does not end when the restored workload exits, and the workload's exit code is lost. The Pod can stay Running and Ready with no workload; the liveness probe is what notices.
- Signals reach the inert command, not the workload. When the Pod is deleted,
the workload gets no
SIGTERM; it is killed at the end of the grace period. - After a failed restore, the agent kills the container. If it starts again, it runs the inert command and is not restored. Delete the Pod to restore again.
- The proxy's exit codes, 112 and 113, do not apply.