[wmousa]

Why

On Kubernetes the storage node reserves huge pages while it is added: the init Job POSTs apply_config back to the node agent, which writes nr_hugepages per NUMA node and restarts the kubelet. Talos cannot take any of that (immutable host, no shell), and the only thing that told the Job so was a sniff of /proc/version for the word talos, after which the node simply skipped. Nothing reserved the pages, and a pre-provisioned pool that was short left the SPDK pod Pending on Insufficient hugepages-2Mi with no message naming the number.

What

Talos is stated the way OpenShift is: TALOS_CLUSTER on the node agent, set by the operator from the deployment's environment (as OPENSHIFT_CLUSTER is). A stated Talos node reserves its huge pages through the Talos API:

• _provision_talos_hugepages checks the host's pool against the pod's hugepages-2Mi request. A pool that already covers it is left alone (no kubelet restart on a plain node restart).
• When short, it renders storage_talos_hugepages_job.yaml.j2: a Job on the node running the official talosctl image (scratch + binary, so one container per command). An init container runs talosctl patch machineconfig --mode=no-reboot with a strategic-merge patch putting the per-NUMA counts into machine.sysfs (devices.system.node.node<N>.hugepages.hugepages-2048kB.nr_hugepages, applied live, merges into whatever else is in machine.sysfs); the main container runs talosctl service kubelet restart so the node's capacity follows. Credentials are the Secret Talos issues for a talos.dev/v1alpha1 ServiceAccount in our namespace, mounted where talosctl looks by default.
• The add then waits until both the host (Hugetlb) and the kubelet (status.capacity["hugepages-2Mi"]) cover the request, and fails with the numbers if either does not. A missing credentials Secret fails fast with the setup steps instead of a Job stuck in ContainerCreating.
• The sizing is the one apply_config has always used, factored into _hugepages_per_numa and shared by both paths, so a Talos host ends up with the same pool as a sysfs-written one.
• The init Job skips the nr_hugepages write and kubelet restart through one MANAGE_HUGEPAGES variable (the kernel sniff now only feeds it, as a fallback), and the CPU-topology / core-isolation Jobs, which nsenter into the host, are skipped with a warning.

Managed hosts are untouched: without the flag nothing Talos-related runs.

Knobs: TALOSCTL_IMAGE (default ghcr.io/siderolabs/talosctl:v1.14.3, pin to the cluster's minor / mirror for air-gapped sites) and TALOS_API_SECRET (default talos-api-access-talos-secrets, the talosctl inject serviceaccount naming).

docs/talos.md has the cluster-side prerequisite (enable KubeTalosAPIAccessConfig / machine.features.kubernetesTalosAPIAccess on the control plane with os:admin for the simplyblock namespace, create the talos.dev ServiceAccount) and the knobs.

Detection and the kubelet CPU topology (added 2026-10-21)

• Talos is detected, not declared. The agent reads status.nodeInfo.osImage of its Node (Talos (v1.12.7)), the same source discovery derives environment: Talos from. TALOS_CLUSTER=true|false is now only an override (a derivative with a different osImage, or forcing the managed path); an older operator or hand-written CRs take the Talos path on their own. The kernel-string sniff inside the init Job stays only as the fallback for that script.
• Kubelet CPU topology through the Talos API. The vanilla CPU-topology Job edits the kubelet config file and restarts the kubelet; on Talos the same four keys (cpuManagerPolicy: static, reservedSystemCPUs, topologyManagerPolicy: single-numa-node, topologyManagerScope: pod) are merged into machine.kubelet.extraConfig by a third talosctl Job. Talos applies it without a reboot and restarts the kubelet itself; the add waits for the node to be Ready again and checks the result. It runs under the usual switches (CPU_TOPOLOGY_ENABLED and not SKIP_KUBELET_CONFIGURATION).
• Idempotent through the kubelet's live config. Before patching, the agent reads nodes/<name>/proxy/configz; a kubelet that already runs with the settings is left alone, since Talos restarts the kubelet on every machine-config change and a restart per SPDK restart would be a regression. Without nodes/proxy get on the role (operator Main lvol sync delete #734 adds it) the step is skipped with a warning for the same reason.
• Verified on the lab cluster, worker-4 (2026-10-21 10:35-10:39 UTC, agent image ...-42d2689d, operator 025137ba). The kubelet there already ran cpuManagerPolicy: static via machine.kubelet.extraArgs but without reservedSystemCPUs, so configz read static / None; the restart created snode-talos-kubelet-config-worker-4 (10:36:43), the merge patch landed in machine.kubelet.extraConfig, Talos restarted the kubelet (SIGTERM/started at 10:36:45), the node stayed Ready, configz read static / 0,1 / single-numa-node / pod and the SPDK pod started at 10:37:04. A second restart logged already runs with the CPU topology settings; nothing to apply, created no Job and did not restart the kubelet. One fix came out of the first attempt: the client's preloaded str for the proxy call is a Python dict repr, not JSON (42d2689 reads the raw body).
• Core isolation (isolcpus etc.) is not applied on Talos: kernel arguments live in machine.install.extraKernelArgs and need an upgrade/reboot cycle. Documented.

Operator side (separate PR, simplyblock-operator)

• Emit TALOS_CLUSTER from spec.environment: Talos: Talos *bool on StorageNodesSpec next to UbuntuHost, workload.Talos = ptr.To(true) in the KubernetesEnvironmentTalos case of expansion.go, {Name: "TALOS_CLUSTER", Value: ptr.StringOrDefault(wl.Talos, "false")} in storage_node_workload.go.
• Pass TALOSCTL_IMAGE through (image mirroring).
• Optionally create the talos.dev/v1alpha1 ServiceAccount in the namespace when the environment is Talos, and grant simplyblock-storage-node-sa get on secrets so the agent's fast-fail check works (today it has no Secret RBAC; the check skips on 403 and the Job reports instead).

Tests

tests/unit/test_talos_hugepages.py (26 tests): the sizing helpers (_hugepages_per_numa equals the apply_config figures, the sysfs patch shape, Kubernetes quantity parsing, Job-name truncation), the Job template (patch then restart, no shell, Secret mount, NODE_IP from status.hostIP), and the handler under TALOS_CLUSTER=true: short pool → Talos Job with the per-NUMA patch, kubelet restart, then the SPDK pod; sufficient pool → nothing applied; missing Secret → fast fail naming the setup; unreadable Secret (403) → proceeds; Job failure → reported with the Job name; pool still short / kubelet not following → add fails with the numbers; image and Secret from env; core-isolation Job skipped; no flag → nothing Talos-related. Plus the init-script tests (flag renders, one gating variable, dash parse and a stated-Talos run making no curl/nsenter call — the dash cases run in CI).

Tested on the lab Talos cluster (2026-10-16)

Talos v1.12.7, Kubernetes v1.35.4, one control plane + four workers (24 vCPU / 24 GiB, 4x QEMU NVMe on vfio-pci, single NUMA). Talos API access enabled on the control plane for os:admin in namespace simplyblock; talos.dev/v1alpha1 ServiceAccount created, Secret issued within seconds.

Mechanism (morning): the rendered Job on worker-1 took the pool 6200 → 6300 pages and the kubelet capacity 12400Mi → 12600Mi in ~30 s, node never left Ready. talosctl patch mc --dry-run confirmed the dotted sysfs key; a JSON 6902 patch is refused on this multi-document config, the merge patch is accepted.

Full node-add flow (afternoon, with operator #734 emitting TALOS_CLUSTER=true, all worker pools cleared to 0 first): discovery detected environment: Talos and the NVMe PCI addresses on its own; all four nodes were added through the Talos Jobs (3057 pages each, hugepages-2Mi 0 → 6114Mi), SPDK pods request 5390Mi, cluster Online/active at 13:01 UTC.

Finding, fixed in the last commit: two of the four adds failed their first attempt with "the kubelet still reports 0 MiB". The single Job patched machine.sysfs in its init container and restarted the kubelet in its main container, but Talos writes the kernel parameter asynchronously, so on worker-1 and worker-3 the kubelet came back before the pages existed and advertised zero; a second kubelet restart fixed it. The flow is now two Jobs with the agent waiting for the host pool in between, each step skipped when its half already holds, and talosctl's output copied into the agent log. Two-Job flow verified (13:25 UTC, agent image ...-36ef2ea9): worker-4 shut down, its pool dropped to 0 and kubelet restarted (capacity 0), then sbctl sn restart: patch Job 13:25:45 → host pool 3057 pages → kubelet Job 13:25:48 → capacity 6114Mi by 13:25:59 → SPDK pod 13:26:09, no manual help. A second restart with the pool in place logged already covers ... nothing to apply and created no Job, so a plain restart never restarts the kubelet.

Also seen: images.talosctl on the deployment config was rejected by ImageSpec's registry allow-list (fixed on the operator branch with ExternalImageSpec), so this run used the agent's default talosctl:v1.14.3 against Talos v1.12.7 — the version skew worked.

Machine-config mode: pre-provisioned by default, managed on request (2026-11-02)

Applying the huge-page pool and the kubelet CPU topology through the Talos API needs os:admin credentials in the simplyblock namespace, which not every site wants to hand out. Both steps now run in one of two modes, chosen by TALOS_MACHINE_CONFIG on the node agent (the operator sets it from storageNodes.talosMachineConfig, which the ClusterDeploymentConfig states as spec.cluster.talosMachineConfig; operator #734):

• preprovisioned (default): the machine configs already carry the pool and the kubelet settings, written by whoever provisions the cluster. The agent only verifies them (host pool via Hugetlb, the kubelet's hugepages-2Mi capacity, the kubelet's live configz) and applies nothing. A node that falls short is refused, and the refusal carries the exact patch to apply with talosctl -n <node ip> patch mc -p @patch.yaml: the per-NUMA machine.sysfs counts, or the machine.kubelet.extraConfig block. A pool that the host has but the kubelet does not advertise yet gets a "restart the kubelet" hint instead of a patch. No Talos API credentials are needed.
• managed: the behaviour this PR had so far: the talosctl Jobs apply the patches, which needs the talos.dev/v1alpha1 ServiceAccount Secret.

An unknown value is treated as pre-provisioned with a warning. docs/talos.md now describes both modes, the pre-provisioning recipe (machine.sysfs + machine.kubelet.extraConfig) and the two pitfalls seen on the lab: the kubelet reads its huge-page capacity at start, so a pool written after boot needs talosctl service kubelet restart; and a kubelet that ran under another CPU-manager policy needs /var/lib/kubelet/cpu_manager_state removed, which the managed mode does by itself.

Tests: TestSpdkProcessStartOnPreprovisionedTalos (8 tests) next to the managed-mode class: the sysfs refusal with the patch, the patch equal to the managed-mode one, the kubelet-lag hint, a sufficient node proceeding without credentials, an unconfigured node, the extraConfig refusal creating no Talos Job, an already configured kubelet proceeding, the unknown-mode fallback, and case-insensitive values.

Lab run (2026-11-02, Talos v1.12.7, four workers, operator #734 at 50de1970)

• Default, nothing stated on the ClusterDeploymentConfig: worker-4, whose machine config already had the settings, came online with "nothing to apply". Worker-1, running cpuManagerPolicy: static but without reservedSystemCPUs / topology manager, was refused with the extraConfig patch in the agent log. Applying that patch verbatim with talosctl patch mc had Talos restart the kubelet within about ten seconds, and the next node-add retry brought the node online without further help.
• Switched to Managed on the live StorageCluster: the DaemonSet re-rendered within seconds, worker-2 was refused naming the missing Secret, and after the talos.dev ServiceAccount was created workers 2 and 3 went through the kubelet-state and kubelet-config Jobs and came online. Cluster Online/active.

Finding, not fixed here: the operator's NodeAddFailing event shows only Node add result: False (the task runner raises TaskRetry with that text, add_node returns a bool), so the refusal with the patch is only in the node agent's log. Propagating the agent's message into the task result would make the pre-provisioned mode usable without reading agent logs.

Rebased on main 5543922.

🤖 Generated with Claude Code

[wmousa]

Rebased onto main (103 commits behind; one test conflict from the port-block removal) and extended with the machine-config mode, on branch fix/talos-hugepages-preprovisioned-r2 pending a force-push of this branch.

What changed: TALOS_MACHINE_CONFIG (preprovisioned default / managed), set by the operator from storageNodes.talosMachineConfig (operator #734). Pre-provisioned: the node checks the pool, the kubelet's advertised capacity and the live kubelet configuration, applies nothing, and fails the add with the exact machine-config patch to apply (sized for the worker, talosctl patch mc form); no Talos API credentials exist anywhere. Managed: the talosctl Jobs as before. docs/talos.md describes both.

Lab (Talos v1.12.7, 4 workers, operator feat-talos-cluster-flag-r2 + this branch's image, 2026-11-02):

• Default mode, no talosMachineConfig stated: worker-4 (kubelet already configured) online with "nothing to apply"; worker-1 refused with the extraConfig patch in the message. Applied that patch verbatim with talosctl -n <ip> patch mc -p @patch.yaml: Talos restarted the kubelet in ~10 s, and the add's next retry brought the node online a minute later.
• Switched the StorageCluster to Managed with the Secret absent: worker-2 refused naming the Secret and the two setup steps. Created the talos.dev ServiceAccount: the next retry ran the kubelet-state and kubelet-config Jobs on workers 2 and 3, both online, cluster active.

One gap seen on the way, not fixed here: the operator's NodeAddFailing event carries only "Node add result: False" (add_node logs the agent's message and returns a bool, tasks_runner_node_add.py:92), so the patch text is only in the agent log. Propagating the agent's error into the task result is a separate change.

🤖 Generated with Claude Code