Restoring starts a new replica from a snapshot instead of cold-starting it. The
restored pods carry the nvidia.com/restore-from annotation, which names the
PodSnapshot to restore from. During pod startup, the node agent restores the
checkpointed state into the container.
-
You need a Ready
PodSnapshot, which Checkpoint a Pod produces with either method. Check that it exists:kubectl get podsnapshot --namespace "$WORKLOAD_NAMESPACE" -
Run the commands from Choose Your Recipe in the same terminal. The commands on this page use the values they set.
Snapshot replaces the container's command with its restore proxy, as the Restore Pod Contract describes, so the application does not start from scratch. The proxy waits for the restore, then follows the restored process. The container ends when the restored process does.
The restored pod also becomes Ready differently from the source pod. The source pod
is Ready when the workload writes the ready-for-snapshot file. The restored pod is
Ready when the framework writes its post-restore file, such as vllm-restore-ready
for vLLM. Each framework page lists its own under After the Restore:
vLLM,
SGLang and
TensorRT-LLM.
A restore profile is a Deployment manifest that creates the restored pod. It keeps the source pod's image, resources, node selection, volumes and security settings. It leaves out what only the source pod needs: the init containers (the model download and, for eight GPUs, the copy of Snapshot's multi-GPU libraries), the container arguments and engine settings, and the multi-GPU launcher and annotation.
Each framework ships two profiles under restore/, one for each GPU count. Your
recipe already chose one: $RESTORE_MANIFEST, set in
Set the Names from your FRAMEWORK and GPUS. The table
lists all of them:
| Framework | 1 GPU | 8 GPUs |
|---|---|---|
| vLLM | 1-gpu.yaml | 8-gpu.yaml |
| SGLang | 1-gpu.yaml | 8-gpu.yaml |
| TensorRT-LLM | 1-gpu.yaml | 8-gpu.yaml |
A profile works for every model of its framework and GPU count, because the
model configuration comes from the snapshot. It also works after either
checkpoint method, PodSnapshot or SnapshotJob. Keep the container image the
same as the source's.
Every profile carries the annotation nvidia.com/restore-from: ${SNAPSHOT_NAME},
a placeholder for the PodSnapshot to restore from.
Open your profile ($RESTORE_MANIFEST) and replace ${SNAPSHOT_NAME} with your
recipe's snapshot name, then save the file. echo "$SNAPSHOT_NAME" prints it, for
example vllm-qwen3-0-6b-snapshot.
$SNAPSHOT_NAME is the name of the PodSnapshot you made when you checkpointed,
whichever method you used:
SnapshotJob: thePodSnapshothas the same name as the jobPodSnapshot: it is the name you gave it in Checkpoint, Option 1
On nodes without NRI v0.10 or newer, there is no restore proxy. Before applying the profile, edit it:
- Change the container's command to
["sleep", "infinity"] - Remove its
args - Add a liveness probe on the framework's API
For details, see the Restore Pod Contract.
-
Apply the profile you edited:
kubectl apply \ --namespace "$WORKLOAD_NAMESPACE" \ --filename "$RESTORE_MANIFEST"
-
Wait for the rollout to finish:
kubectl rollout status \ --namespace "$WORKLOAD_NAMESPACE" \ deployment/"$RESTORED_NAME" \ --timeout="$WAIT"
The node agent adds a nvidia.com/Restored condition to the pod once the restore
completes. Watch it, along with pod readiness, to confirm:
kubectl get pods --namespace "$WORKLOAD_NAMESPACE" \
--selector app="$RESTORED_NAME" \
--output jsonpath='{.items[0].status.conditions[?(@.type=="nvidia.com/Restored")].status}'It prints True. If the restore does not complete, see
Troubleshoot Restores.
A pod gets one restore. The supported frameworks use a Deployment, which restarts
containers, so if the restored process exits, the container ends in
CrashLoopBackOff instead of starting again. Delete the pod to restore a new
replica.
In practice, higher-level systems create these restored Deployments rather than
applying them by hand. To generate restore pods from a controller, operator or
serving platform, implement the
Restore Pod Contract. It specifies the
required annotations, control volume and SNAPSHOT_CONTROL_DIR, plus the optional
startup gate and seccomp profile a restored pod may carry.