Audience: Operators
This page describes the lifecycle of a PhysicalHost — the states it moves through, what triggers each transition, and how to recover when it gets stuck.
The state constants are defined in api/v1beta2/physicalhost_types.go. The transitions are driven by controllers/physicalhost_controller.go and the Beskar7Machine reconciler in controllers/beskar7machine_controller.go.
| Constant | Status string | Meaning |
|---|---|---|
StateNone |
"" |
A new host, until its first successful BMC contact moves it to Available (a failed one moves it to Error). |
StateUnknown |
"Unknown" |
Defined in the API but never set by the controller. |
StateEnrolling |
"Enrolling" |
Defined in the API but never set by the controller. |
StateAvailable |
"Available" |
BMC reachable; no consumer claim. Eligible for a Beskar7Machine to claim. |
StateInUse |
"InUse" |
A Beskar7Machine has claimed the host (Spec.ConsumerRef is set). |
StateInspecting |
"Inspecting" |
The inspection image is booting / running on the host. |
StateDeploying |
"Deploying" |
Hardware inspection passed and the inspector is writing the OS image to disk. Entered when the inspection report is accepted; left through the inspector's provisioned or provision-failed callback. The deployment timeout fails only the Beskar7Machine and does not signal the host. |
StateReady |
"Ready" |
OS deployment complete (the inspector's provisioned callback was received). Beskar7Machine.Spec.ProviderID, Status.Ready, and Status.Initialization.Provisioned are set at this point. |
StateError |
"Error" |
A terminal-or-recoverable error condition. See Status.ErrorMessage. An unreachable BMC (RedfishConnectionReady reason BMCUnreachable) is the kind that clears by itself. A failed provisioning run — the inspector's /provision-failed report or an inspection timeout — is the kind that does not: a claimed host keeps that Error until it is released. |
The legacy Claimed, Provisioning, Provisioned, Deprovisioning strings from v0.3 are gone. The Go code previously kept deprecated alias constants (StateClaimed = "InUse", etc.); those aliases have been removed. Code that imported them must switch to the canonical state constants above.
(object created)
│
▼
Available ◀───────────────────────────────────────┐
│ │
Beskar7Machine │ sets spec.consumerRef │ Beskar7Machine
(atomic claim) │ │ deletion releases
▼ │ ConsumerRef
InUse │
│ │
Beskar7Machine │ patches inspection-request="inspect" │
▼ │
Inspecting ── inspection timeout ──► Error ¹ │
│ │
PhysicalHost │ inspection-result ConfigMap consumed, │
controller │ report validated (inspection-complete signal)│
▼ │
Deploying ── POST /provision-failed ² ─► Error ¹ │
│ │
inspector │ POSTs /api/v1/provisioned (202) ³ │
▼ │
Ready ───────────────────────────────────────► (deletion)
┌───────────────────────────┐
│ Error (any state) │ recovers when the underlying
└───────────────────────────┘ problem clears (e.g. BMC reachable)
¹ The run failed. A claimed host keeps this Error, through any later BMC
failure, until Beskar7Machine deletion releases it.
² Applied before the host tries its BMC, so a BMC failure cannot get in
first. A report that arrives while the host is still Inspecting is kept
until the host is Deploying.
³ Applied before the host tries its BMC as well, and kept the same way on a
host still Inspecting. A /provision-failed report waiting with it wins.
| From | To | Trigger | Where |
|---|---|---|---|
| (no state) | Available |
First successful Redfish reconcile, no ConsumerRef. |
physicalhost_controller.go:reconcileNormal |
Available |
InUse |
A Beskar7Machine writes Spec.ConsumerRef. |
beskar7machine_controller.go:findAndClaimOrGetAssociatedHost |
InUse |
Inspecting |
Beskar7Machine writes inspection-request: inspect annotation. It is applied only to a host that is InUse; on a host in any other state it is removed and nothing changes, so a duplicate never takes an Inspecting or Deploying host back to the start of its run. A host that went to Error about its BMC just after the machine booted it drops the request, and the machine sends it again once the host is back InUse; an inspection report that arrived meanwhile is kept (InspectionPhase stays Complete). |
beskar7machine_controller.go:triggerInspection, physicalhost_controller.go:applyInspectionRequest |
Inspecting |
Deploying |
PhysicalHost controller consumes the inspection-result ConfigMap, validates the report, and advances the phase. Beskar7Machine then drives the state forward via inspection-request: inspect-complete. |
physicalhost_controller.go:applyInspectionResultAnnotation, beskar7machine_controller.go:validateInspectionReport |
Inspecting |
Error |
Inspection timeout (--inspection-timeout, default 10 min) — Beskar7Machine writes inspection-request: timeout, and the host records Inspection timed out. The run has failed: the host keeps this Error until it is released. |
beskar7machine_controller.go:handleInspectingHost, physicalhost_controller.go:applyInspectionRequest |
Deploying |
Ready |
Inspector POSTs POST /api/v1/provisioned/{ns}/{host} (§4.4 of docs/inspector-contract.md). The provisioned handler patches ProvisionedRequestAnnotation; the PhysicalHost reconciler transitions to Ready, and clears the annotation on a later pass, once status shows Ready — the reconciler's patch writes metadata before status, so clearing it in the same pass would lose the report to a status write that fails. It applies the report before it tries the BMC, so a deployment that finishes during an outage provisions the machine rather than timing it out. A /provision-failed report waiting on the same host is applied instead, the conservative choice when both are present (one inspector run never sends both). |
controllers/provisioned_handler.go, physicalhost_controller.go:applyProvisionedRequestAnnotation |
Inspecting (claimed) |
unchanged, then Deploying and Ready |
/provisioned arrives before the host is Deploying: the inspector does not wait for the host (§9.2), and the host goes to Deploying only once the machine has validated the inspection report — which can come after the whole deployment when the controllers were away while a callback-only instance kept answering the inspector. The host keeps the annotation, and applies it in the pass after inspect-complete has moved it to Deploying. It clears the annotation without a transition if the host goes anywhere else — the machine rejects the hardware (HardwareRequirementsNotMet), the host is released — or if the report came before the run's inspection report. |
physicalhost_controller.go:applyProvisionedRequestAnnotation |
Deploying |
Error |
Inspector POSTs POST /api/v1/provision-failed/{ns}/{host} (§4.5 of docs/inspector-contract.md). The provision-failed handler patches ProvisionFailedRequestAnnotation; the PhysicalHost reconciler transitions to Error with the inspector's sanitized reason in Status.ErrorMessage and clears the annotation. It does so before it tries the BMC, so the report lands during an outage or a BMC failure that needs a fix, and the machine fails with DeploymentFailed rather than a BMC reason. The run has failed: the host keeps this Error until it is released. |
controllers/provision_failed_handler.go, physicalhost_controller.go:applyProvisionFailedRequestAnnotation |
Inspecting (claimed) |
unchanged, then Error |
/provision-failed arrives before the host is Deploying: the inspector starts Phase 2 as soon as it has posted its inspection report (§9.2), and a fast failure — a target image URL that answers 404 — reports before the Beskar7Machine has validated the inspection report. The host keeps the annotation and applies it once inspect-complete has moved it to Deploying. It clears the annotation without a transition if the host goes anywhere else — the machine rejects the hardware (HardwareRequirementsNotMet), the host is released — or if the report came before the run's inspection report. |
physicalhost_controller.go:applyProvisionFailedRequestAnnotation |
Error about the BMC, during deployment (claimed) |
Error of the failed run |
/provision-failed arrives for a host that v0.8.0 or earlier put in Error over Deploying for a BMC failure that needs a fix (credentials, certificate, TLS config, no ComputerSystem); the current controller keeps such a host Deploying (rows below), so only a host upgraded in that state reaches this row. The inspector does not need the BMC, so the report still applies; the host takes the inspector's reason and keeps it until it is released. |
physicalhost_controller.go:deployInterruptedByBMCError |
Error about the BMC, during deployment (claimed) |
Ready |
/provisioned arrives for the same upgraded host. The inspector finished deploying without the BMC, so the report applies like one on a Deploying host: the host goes to Ready, drops the BMC's Status.ErrorMessage (RedfishConnectionReady keeps reporting the BMC), and stays Ready once the BMC answers instead of going back to InUse, where its machine would boot the inspector again. A /provision-failed report waiting with it wins. |
physicalhost_controller.go:applyProvisionedRequestAnnotation |
Deploying |
unchanged | Deployment timeout (--deployment-timeout, default 20 min, measured from Status.DeployingTimestamp). Only the Beskar7Machine fails (DeploymentTimedOut); the timeout does not signal the host. |
beskar7machine_controller.go:handleDeployingHost |
| any | Error |
Credentials missing, credentials the Secret does not authorise for the address (CredentialsNotAuthorized), TLS-config conflict, Redfish connection or query failed, or an unreachable BMC — on a host that is unclaimed or InUse. A claimed host that is Inspecting, Deploying or Ready, or in the Error of a failed run, keeps its state (rows below). |
physicalhost_controller.go:reconcileNormal, bmc_access.go:resolveBMCAccess |
Inspecting / Deploying / Ready (claimed) |
unchanged | BMC unreachable, or another Redfish/BMC-level fault that needs a fix — missing or refused credentials, credentials the Secret does not authorise for the address, a certificate the client rejects, a malformed address, a BMC with no ComputerSystem, the insecureSkipVerify/CA-bundle conflict. The host keeps its state — the inspector and the installed OS do not need the BMC — and RedfishConnectionReady=False reports the fault (BMCUnreachable, CredentialsNotAuthorized, MissingCredentials, InsecureCABundleConflict, CABundleFetchFailed, RedfishConnectionFailed, or RedfishQueryFailed) until it clears. Provisioning goes on meanwhile: the host still acts on the annotations in the table below, so it moves through its run as it would with the BMC answering. |
physicalhost_controller.go:retryTransientRedfishFailure, physicalhost_controller.go:setConnectionError, physicalhost_controller.go:applyRunAnnotations |
Ready (claimed) |
unchanged | An inspection-request annotation is consumed without effect. The Beskar7Machine can send inspect-complete a second time after reading a copy of the host from before its first request was applied, and a host that kept a /provisioned report while Inspecting is Ready one pass after it is Deploying. |
physicalhost_controller.go:applyInspectionRequest |
Error from a failed run (claimed) |
unchanged | Any BMC failure, and the BMC's recovery. RedfishConnectionReady reports the BMC; Status.ErrorMessage keeps the run's reason. An inspection-request annotation is consumed without effect — the Beskar7Machine can send inspect-complete a second time after reading a copy of the host from before its first request was applied. |
physicalhost_controller.go:provisioningRunFailed, physicalhost_controller.go:applyInspectionRequest |
Error |
Available, or InUse if claimed |
The underlying error clears (BMC reachable again, secret fixed or annotated, TLS config fixed). Not the Error of a failed run, which a claimed host keeps until it is released. |
physicalhost_controller.go:reconcileNormal |
InUse / Inspecting / Deploying / Ready / Error |
Available |
Beskar7Machine deletion clears Spec.ConsumerRef. |
beskar7machine_controller.go:reconcileDelete |
Two Beskar7Machines can race to claim the same Available host. The race is resolved server-side:
- The reconciler lists
PhysicalHostfiltered by thestatus.statefield index forAvailable. - It selects the first host with
ConsumerRef == nil. - It patches the host with
MergeFromWithOptions(base, MergeFromWithOptimisticLock{}). The optimistic-locking option means the patch carries the host's currentresourceVersion, so a concurrent claim from another reconciler fails fast with409 Conflict. - The losing reconciler gets the conflict, requeues, re-lists, and either picks a different host or returns empty.
This means exactly one Beskar7Machine succeeds per host. There is no separate state for "claim in flight" — the claim is the patch.
The Beskar7Machine reconciler signals work to the host through annotations on Spec (never via Status); the PhysicalHost reconciler is the sole writer of the host's status. This pattern (decision D-005 in .claude/context/PROJECT_CONTEXT.md) ensures every controller owns its own resource's status.
Annotations consumed by the PhysicalHost reconciler. None of them needs the BMC, so the reconciler acts on them in every pass: when the BMC answers, when it cannot be reached at the network level, and when it fails in a way that needs a fix (credentials, certificate, TLS config, no ComputerSystem). The inspector's two reports are applied first, before the BMC is tried:
| Annotation | Producer | Consumer action |
|---|---|---|
infrastructure.cluster.x-k8s.io/inspection-request |
Beskar7Machine controller |
Only with its binding (below). Drive Status.State and Status.InspectionPhase. Values: inspect, inspect-complete, timeout. |
infrastructure.cluster.x-k8s.io/bootstrap-url |
Beskar7Machine controller |
Persist URL to Status.Bootstrap.URL. |
infrastructure.cluster.x-k8s.io/bootstrap-token, infrastructure.cluster.x-k8s.io/boot-nonce |
Nothing (retired, D-029) | Removed on sight, never read. The callback credentials live only in the <host>-bootstrap-token Secret; the reconciler mirrors its hashes and expiries into Status.Bootstrap.{TokenHash,IssuedAt,ExpiresAt,BootNonceHash,BootNonceExpiresAt} for operators, and follows it through a Secret watch. The consume record is the /boot handler's. |
infrastructure.cluster.x-k8s.io/inspection-result-ref |
Inspection HTTP handler | Only with its binding (below). Read referenced ConfigMap, persist the InspectionReport, mark HostInspected=True, delete the ConfigMap. |
infrastructure.cluster.x-k8s.io/provisioned-request |
ProvisionedHandler (HTTP) |
Only with its binding (below). Transition Status.State to Ready (D-015), from Deploying or from an Error a BMC failure wrote over it. Value: "provisioned". Applied before the BMC is tried, and after provision-failed-request, which wins when both are present. On a claimed host still Inspecting, left in place until the host is Deploying. Cleared on the pass after the action, once status shows Ready, or without one when the host cannot be provisioned by it. |
infrastructure.cluster.x-k8s.io/provision-failed-request |
ProvisionFailedHandler (HTTP) |
Only with its binding (below). Transition Status.State to Error and persist the value — the sanitized reason, prefixed inspector reported deploy failure: — to Status.ErrorMessage (contract v4.1), from Deploying or from an Error a BMC failure wrote over it. Applied before the BMC is tried. On a claimed host still Inspecting, left in place until the host is Deploying. Clear after action, or without one when the host cannot be failed by it. |
Anyone allowed to patch PhysicalHost objects could write the three annotations the inspector's callbacks leave, and push a host to Ready, inject a hardware report or fail a run with no inspector involved; or the inspection-request the Beskar7Machine controller writes, and record an inspection timeout (which fails the machine at once) or send inspect-complete (which takes the host to Deploying, where it ends in DeploymentTimedOut) with no machine involved. Each of the four therefore travels with a sibling annotation named <annotation>-binding (inspection-result-ref-binding, provisioned-request-binding, provision-failed-request-binding, inspection-request-binding), written in the same patch (decisions D-034, D-037, SEC-15). The binding is an HMAC-SHA256, keyed by the host's bearer token, over the annotation's key and value, the host, the machine and boot nonce of the claim the token was minted for, and, for inspection-result-ref, the SHA-256 of the report.json the ConfigMap holds. A callback handler can compute it because the bearer middleware has just authenticated the caller with that token. The Beskar7Machine controller mints the token, so it signs with the credentials in the Secret: an inspect with the ones it wrote in that same pass, an inspect-complete or timeout with the ones the Secret holds then. It writes no request it cannot sign.
The reconciler recomputes the binding from the <host>-bootstrap-token Secret, so forging one takes the Secret, the same bar as reading the host's bootstrap data. It reads the Secret from the API server, not from its cache, because the machine signs an inspect in the pass that mints or rotates the credentials, and a cache that had not seen the write yet would show the credentials before it. It acts on an annotation only if its binding matches. An annotation with no binding, a wrong binding, a binding that is for another value (a genuine inspect edited into a timeout), a binding from an earlier boot cycle or claim (the nonce or the machine's UID has changed), or an inspection-result-ref whose ConfigMap was rewritten after the handler bound it is removed, by key, and does nothing else: no state transition, no status write, and its ConfigMap is not deleted. The reconciler logs the host and the annotation key at Info, and never the token, the binding or the value. Token expiry is not checked then: the writer held a valid token when it wrote the annotation, and the token is cut to 5 minutes at Ready while a /provisioned report may still be waiting. If the Secret cannot be read at that moment, the annotation waits for the next pass.
The inspector sends exactly what it always did. The controller and any callback-only instance have to run the same version, since an annotation written by an older release has no binding. A request the older controller had written and the reconciler had not applied yet is removed, and the Beskar7Machine controller writes it again, signed, in its next pass: inspect while the host is InUse, inspect-complete while the host is Inspecting with its report in. A timeout is not written again, since the machine has failed for it already and stays Failed; its host keeps its state until it is released. Upgrade with no host Inspecting or Deploying all the same (Upgrading).
bootstrap-url is written by the Beskar7Machine controller and carries no binding. A hand-set inspect-complete is ignored; a signed one takes a host to Deploying, and no further: Ready takes a bound provisioned-request.
The reconciler cannot complete the first BMC handshake: the host's state stays "" or goes to Error.
kubectl describe physicalhost <name>Look at the RedfishConnectionReady condition reason — it is one of BMCUnreachable, CredentialsNotAuthorized, MissingCredentials, InsecureCABundleConflict, CABundleFetchFailed, RedfishConnectionFailed, or RedfishQueryFailed. Fix the credentials Secret, its annotations or the BMC address; the next reconcile transitions to Available.
Three retry cadences sit behind that. A host whose BMC is simply unreachable — reason BMCUnreachable, message BMC unreachable (connection refused) or similar, covering a refused or reset connection, no route, a DNS failure, a timeout, or a 502/503/504 from a BMC that is still booting — is retried every 15 seconds, flat, and enrols on the first attempt after the BMC answers. Nothing has to be fixed for that to clear, and a Beskar7Machine holding the host waits for it instead of failing (see Beskar7Machine → A BMC outage is not a terminal failure). A host refused with CredentialsNotAuthorized (its credentials Secret does not authorise the address, so nothing was sent) or InsecureCABundleConflict is re-checked every 5 minutes, since only an edit clears it; a Beskar7Machine holding a host refused with CredentialsNotAuthorized waits for that edit instead of failing. Every other Redfish failure needs a change to the spec, the Secret or the BMC itself, so it backs off exponentially (5s, 10s, 20s, … capped at 30 minutes) and a host that has been failing for a while can take a few minutes to notice the fix. Editing the PhysicalHost or its credentials Secret, annotations included, wakes the controller immediately.
The host booted but the inspection image never POSTed a report. Check:
kubectl get physicalhost <name> -o jsonpath='{.status.inspectionPhase}'
kubectl get physicalhost <name> -o jsonpath='{.status.inspectionTimestamp}'If inspectionTimestamp is more than 10 minutes ago, the Beskar7Machine controller will mark InspectionTimedOut (terminal) and write inspection-request: timeout, which moves the host to Error for as long as it stays claimed. To recover, fix the underlying iPXE / inspection-image / network issue, then delete-and-recreate the Beskar7Machine (which deletes the BMC token Secret, clears ConsumerRef, and lets the host return to Available).
The host entered Deploying (inspection passed and the inspector is writing the OS image) but the provisioned callback never arrived. Check Status.DeployingTimestamp:
kubectl get physicalhost <name> -o jsonpath='{.status.deployingTimestamp}'If the timestamp is older than the --deployment-timeout (default 20 min), the Beskar7Machine controller marks the Beskar7Machine terminally failed: status.phase=Failed and InfrastructureReady=False with reason DeploymentTimedOut. The timeout does not signal the host. An inspector that can still report a failure calls /provision-failed instead, which moves the host to Error at once, whether or not its BMC is reachable (see Stuck in Error). Its /provisioned report does not need the BMC either, so a BMC outage alone does not time out a deployment the inspector reported finished. Common causes of a timeout:
- The OS image download is slow or stalled — check network reachability from the host to
Beskar7Machine.Spec.TargetImageURL. - The inspector's TLS verification failed for the provisioned-callback endpoint — check that
beskar7.apiis externally reachable and that the certificate uses a two-tier PKI (CA cert distinct from the server cert; see the TLS note indocs/inspector-contract.md§8). - The inspector aborted after the disk write (e.g. digest mismatch,
COS_OEMmount failure) — check the host's serial console or BMC event log.
To recover, delete and recreate the Beskar7Machine. This clears ConsumerRef, releases the host to Available, and starts a fresh provision cycle with new nonce and token.
Status.ErrorMessage is the source of truth. Common cases:
redfishConnection.insecureSkipVerify=true is mutually exclusive with caBundleSecretRef: edit the spec — pick one. The reconciler resumes once the spec is valid.failed to retrieve credentials: …: the named Secret does not exist, cannot be read, or is missingusername/password. Create or fix it.BMC credentials not sent: …: the credentials Secret does not authorise this host's address (RedfishConnectionReady=False (CredentialsNotAuthorized)), so nothing was sent to the BMC. The message names what is missing. Check that the address really is this host's BMC, then add it to the Secret'sbeskar7.infrastructure.cluster.x-k8s.io/bmc-addressesannotation (and setbmc-insecure-transport: "true"for anhttp://address orinsecureSkipVerify: true). ABeskar7Machineholding the host reportsInfrastructureReady=False (WaitingForBMC)meanwhile and carries on once the Secret is annotated. See Troubleshooting → 17.BMC unreachable (…); retrying every 15s: nothing to fix on the object. The host retries and leavesErroron the first attempt that connects; check the network path from the controller to the BMC if it does not. ABeskar7Machineholding the host reportsInfrastructureReady=False (WaitingForBMC)meanwhile and carries on afterwards — it is not failed and does not need replacing.inspector reported deploy failure: …: the inspector's/provision-failedreport, with its reason after the colon (no details providedwhen it sent none). The run failed, and theBeskar7Machineis failed withDeploymentFailed, quoting the message. The host keeps thisErrorwhile it is claimed — a later BMC failure or recovery changes onlyRedfishConnectionReady— and returns toAvailableonce theBeskar7Machineis deleted and releases it.Inspection timed out: see above. Kept the same way until the host is released.
If the BMC is permanently unreachable and you need to delete the consuming Beskar7Machine without waiting for the Redfish power-off / boot-clear, set the force-release annotation on the Beskar7Machine before deleting it:
kubectl annotate beskar7machine <name> \
infrastructure.cluster.x-k8s.io/force-release=true
kubectl delete beskar7machine <name>The Beskar7Machine controller skips the Redfish cleanup, clears ConsumerRef, and removes the finalizer. The host returns to Available.
If a finalizer is genuinely stuck (and only after exhausting the recovery paths above), you can remove it manually. This is destructive — it leaves Redfish state untouched.
kubectl patch physicalhost <name> --type=merge -p '{"metadata":{"finalizers":[]}}'kubectl describe physicalhost <name> shows the conditions list — native metav1.Condition, no severity field, every condition (including True) carries a reason. Key types:
RedfishConnectionReady— BMC connectivity. True reasonRedfishConnected;False (BMCUnreachable)while the BMC cannot be reached, the oneFalsereason that clears by itself.HostAvailable— no consumer holds the host. True reasonHostAvailable;False (HostClaimed)whilespec.consumerRefis set.HostInspected— inspection report has been persisted. True reasonHostInspected;False (HostReleased)when a host returns toAvailableafter a run.
Full reason lists: PhysicalHost → Conditions.
kubectl get events --field-selector involvedObject.kind=PhysicalHostThe controller emits two: AdoptedProvisionedClaim (Normal) when a host whose machine already holds it by providerID rebuilds its Ready state, as after clusterctl move, and DeletingClaimedHost (Warning) when a host is deleted while still claimed. Transitions otherwise show in the conditions.
See Metrics for the Prometheus surface. The relevant metric for state observation is beskar7_controller_physicalhost_states_total{state=...}.