Skip to content

Latest commit

 

History

History
194 lines (137 loc) · 7.18 KB

checkpoints.md

File metadata and controls

194 lines (137 loc) · 7.18 KB

Troubleshoot Checkpoints

Use this page when a checkpoint does not finish. It starts with a PodSnapshot that never becomes Ready. Then it lists each step a checkpoint can fail at, with the likely causes and what to check. To find which step failed, see Reading a Failure Message.

A Snapshot Never Becomes Ready

A PodSnapshot becomes Ready only after the snapshot-agent confirms the snapshot contents. Writing the snapshot is not enough on its own. Check the status and the agent logs:

kubectl get podsnapshot <name> -n <ns>
node=$(kubectl get pod <source-pod> -n <ns> -o jsonpath='{.spec.nodeName}')
agent=$(kubectl get pods -n <snapshot-ns> -l app.kubernetes.io/component=snapshot-agent \
  --field-selector spec.nodeName=$node -o name)
kubectl logs $agent -n <snapshot-ns> --all-containers

The agent that handled the checkpoint is the one on the source pod's node; reading the DaemonSet picks an arbitrary pod. Reading a failure message explains the lookup.

A common cause is a replica manifest that does not follow the framework guides -- missing the control volume, seccomp profile, or readiness gate Snapshot relies on -- or that omits mounts or secrets the replica needs to start.

When the container has a checkpointReadinessProbe, the agent waits for that probe, not for Ready, and the source pod has a WaitingForCheckpointReadiness event. The agent logs each failed check at debug level. Check a file probe from inside the source container. Check an HTTP probe against the pod's IP and port from another pod or the node, not against localhost inside the container: the agent sends the request from the node to the pod's IP. The wait has no deadline of its own, so a probe that never passes waits until you delete the PodSnapshot, or until the source Pod ends: it fails, exits or is deleted, or a SnapshotJob's activeDeadlineSeconds stops it. The PodSnapshot then fails at most five minutes after the Pod ended.

A Checkpoint Fails with InvalidCheckpointReadinessProbe

The agent refused the container's checkpointReadinessProbe before running it, because it can never pass: the container has no port with the probe's port name, or the probe's HTTP path does not parse as a URL path. The source pod also gets a Warning event with this reason.

The PodSnapshot's spec cannot change, so create a new one with a fixed probe. If a SnapshotJob created it, create a new SnapshotJob with the probe fixed in podSnapshotTemplate.containers instead: the SnapshotJob copies its template into the PodSnapshot it creates.

Checkpoint Fails at a Step

resolving checkpoint artifact path

The agent computes where the artifact will live under the storage base path from the content UID and container name.

Likely causes:

  • A PodSnapshotContent with an empty or malformed content UID
  • A misconfigured storage base path

inspecting source container

The agent resolves the container's PID and OCI spec through the node runtime, reads its rootfs, overlay upper directory, mounts and network namespace, and discovers its CUDA processes and GPUs.

Likely causes:

  • The container exited before the agent reached it
  • The runtime uses a snapshotter without an overlay upper directory
  • CUDA processes sit in nested PID namespaces
  • GPU discovery failed inside the container
kubectl get pod <source-pod> -n <ns> -o jsonpath='{.status.containerStatuses}'

preparing PageBroker checkpoint

The agent asks PageBroker to open a transaction and provide the staging or destination directory. The cause carries PageBroker's failure code, such as INSUFFICIENT_STORAGE or STORAGE_ERROR, or a connection error.

Likely causes:

  • The pagebroker container is not ready
  • The snapshot volume is full or unavailable
kubectl logs $agent -n <snapshot-ns> -c pagebroker
kubectl get pvc -n <snapshot-ns>

inspecting cuinterpose

When the workload opted into the cuinterpose shim, every CUDA process must have loaded it, its delivery mount must be read-only, and the coordinator must answer before the checkpoint becomes destructive.

Likely causes:

  • A CUDA process started without the shim
  • The delivery mount is writable
  • The coordinator is not running

staging CUDA job file

For a driver-managed GPU checkpoint, the agent copies the CUDA checkpoint job file out of the container and checks that it lists every GPU.

Likely causes:

  • The workload has not reached its quiescent point, so the job file is missing or stale
  • The GPU count in the file differs from the devices discovered

configuring CRIU dump and writing checkpoint manifest

The agent builds CRIU's dump options from the inspected container and writes manifest.json into the staging directory.

Likely causes:

  • A mount layout CRIU options cannot express
  • A staging directory that is not writable

checkpointing GPU state

The agent checkpoints GPU memory for each CUDA process, through the driver's checkpoint tool or through PageBroker for CustomStorage, after preparing cuinterpose when the shim is in use. A failure here, or in the CRIU dump that follows, leaves the source process unsafe to resume, so the agent terminates it.

Likely causes:

  • A kernel still running or a stream still busy when the lock was taken
  • A driver without checkpoint support for this GPU
  • Another process holding the device, such as a GPU monitoring agent

dumping with CRIU

CRIU dumps the process tree into the staging directory. The cause is the error CRIU logged.

Likely causes:

  • A file descriptor CRIU cannot dump, such as an io_uring instance when the seccomp profile from the supported frameworks is missing, a socket to a peer outside the container, or an open file on a volume the checkpoint did not record
  • Missing ptrace permission

The agent log carries the CRIU error. Check the source pod against the supported frameworks.

capturing rootfs diff

After the dump, the agent records the container's upper directory as a rootfs diff, keeping every file the dump references even when an exclusion pattern matches it. Other diff problems are logged and the checkpoint goes on without them; the dump has already stopped the source, so a failure here means re-taking the checkpoint from a fresh replica.

Likely causes:

  • The dump's image files cannot be read back from the staging directory
  • A file the dump references is missing from the upper directory, for example one deleted or replaced between the dump and the diff
kubectl logs $agent -n <snapshot-ns> -c agent | grep -i 'rootfs diff'

committing PageBroker checkpoint

PageBroker publishes the staged artifact to its final location. The dump has already stopped the source process at this point, so a failure here means re-taking the checkpoint from a fresh replica.

Likely causes:

  • The final destination ran out of space or became unavailable
  • PageBroker restarted (TRANSACTION_NOT_FOUND)
  • The transaction expired
kubectl logs $agent -n <snapshot-ns> -c pagebroker