Dev zone (Cherryvale, KS) not provisioning — VMs stuck in 'pending' indefinitely

[Amperstrand]

Summary

New VM orders in the Dev VPS zone (Cherryvale, Kansas) are accepted but never leave provisioning_state=pending. They do not transition to provisioning, never reach ready, and never receive an IP. The NVMe zone (Katy, Texas) is healthy as a control, so this looks specific to the Cherryvale site / Dev VPS scheduler, not the API or account.

Reproduction

Two separate runs today (2026-07-31), ordering the smallest Dev VPS:

pkg 80 (Dev VPS - Starter), pricing_id 241, order_form_id 11, $0.24/day

Via SHCClient.submit_order(...) — order succeeds in ~6s, returns a service_id, but get_vm(sid).provisioning_state stays pending / service_status=pending forever.

Evidence

Run	sid	Zone	pkg	Ordered	Final state after wait	Wait	Outcome
1	1911	Cherryvale, KS	80	21:31:20	pending/pending	300s	timeout → cancel
2	1912	Cherryvale, KS	80	21:38:23	pending/pending	600s+	timeout → cancel
2	1913	Katy, TX (control)	23	21:38:26	ready/active, IP 23.182.128.217	~100s	✅ working → cancel

Both Dev-zone orders were cancelled immediately on timeout (clean, prorated). NVMe control VM provisioned normally and was cancelled once ready.

Expected vs actual

• Expected: pending → provisioning → ready within the documented ~100s, like the Katy TX control.
• Actual: Dev-zone orders stall at pending for 10+ minutes with no state change and no IP assignment. No error is surfaced through the API.

Impact

• Blocks the ansible-ease.yml weekly E2E (runs against the SHC Dev VPS).
• Blocks ephemeral GitHub-runner provisioning on Dev VPS plans (docs/github-ephemeral-runners.md, docs/firecracker-pool-mode.md — both rely on pkg 80–84).
• The vm session fixture in tests/conftest.py (pkg 81) will pytest.skip after 300s, masking real regressions behind a skip.

Ask

1. Is this a known Cherryvale site outage / host-scheduler issue on the SHC side?
2. Is there a workaround? e.g. a different Dev package, a node pin, a status channel to check before ordering, or an ETA if the site is being drained.
3. Should the toolkit add a Dev-zone health probe (e.g. order + ready-within-120s smoke check) so CI fails loudly instead of silently skipping?

Environment

• shc-toolkit v2.4.15.1, REST transport (SHC_TRANSPORT=rest)
• API v2.4.15 reachable; account balance OK; orders accepted and cancellable cleanly
• Reproduced from main (commit fdf79f3)

[Amperstrand]

Definitive test: NOT a template issue — the Dev zone scheduler is independently broken

Ran the controlled test suggested in this session: ordered a Dev VPS Starter (pkg 80) in Cherryvale KS with debian12-cloud — the template #24 recommends as a workaround.

Test	Template	Zone	Result
Original #28	debian13-cloud (default)	Cherryvale KS Dev	❌ stuck pending 10min
This test (sid 1980)	debian12-cloud (#24 workaround)	Cherryvale KS Dev	❌ stuck pending 6min
#28 control	debian13-cloud	Katy TX NVMe	✅ ready ~100s

debian12-cloud made zero difference — same pending hang, no IP, never transitions. The #24 workaround does not help here.

Conclusion

#24 and #28 are related but distinct root causes:

• VM cloud-init deadlock on debian13-cloud template — SSH never starts #24 = debian13-cloud template cloud-init deadlock (VM does provision, SSH fails) — a template bug
• Dev zone (Cherryvale, KS) not provisioning — VMs stuck in 'pending' indefinitely #28 = Cherryvale KS scheduler/provisioning hang (VM never provisions, any template) — a zone bug

They share a zone and an overlapping timeline (both since ~2026-07-19/20), but this test proves they are not the same issue. The zone-level failure is upstream of the template-level failure.

Entanglement

#24's workaround (use debian12-cloud) is currently unverifiable: the Dev zone scheduler (#28) prevents any Dev VM from provisioning, so we can't confirm whether the debian13-cloud cloud-init deadlock is still happening. The zone bug masks the template bug. If/when #28 resolves, re-test debian13-cloud to check whether #24 is also still live.

VM 1980 was cancelled immediately on timeout.

[Amperstrand]

Actionable: how to test, what to check, what "fixed" looks like

One-command re-test

Once SHC indicates the Cherryvale site is fixed, verify in one line:

python3 scripts/dev-zone-probe.py                  # default: debian12-cloud, 300s timeout
python3 scripts/dev-zone-probe.py --template debian13-cloud   # also test the #24 template

Exit 0 = zone healthy (VM reached active + IP within the timeout). Exit 1 = still broken. Always cancels the test VM, even on interrupt. Probe added in PR #35.

Exact reproduction (for SHC support)

The failing call sequence — all via the public User API (https://blesta.sovereignhybridcompute.com/user-api/v2):

POST /ordering/submit
  { "package_id": 80, "pricing_id": 241, "order_form_id": 11,
    "package_group_id": 14, "module_group_id": 7,
    "hostname": "devprobe-test", "template": "debian12-cloud" }
→ 201 { "service_ids": [<sid>] }            # order ACCEPTED, billing entry created

GET /vm/<sid>
→ { "provisioning_state": "pending", "service_status": "pending", "ips": [] }
# ...stays pending indefinitely. Never transitions to "provisioning".
# Never receives an IP. Never reaches "active".

Reproduced 3× today (sids 1911, 1912, 1980) across two templates (debian13-cloud default + debian12-cloud workaround). The NVMe zone (Katy, TX) with the same flow provisions in ~100s — so this is isolated to the Cherryvale KS site, not the API, the account, or the billing flow.

What SHC support should check (Cherryvale KS site)

1. Is the Proxmox node online and accepting VM creation tasks? The order creates a billing service entry (confirmed — GET /vm lists it) but the VM never appears on a hypervisor. This points at the Proxmox qm create task either not being dispatched or hanging.
2. Are there stuck tasks in the Proxmox task log for the Dev VPS resource pool (pvesh get /nodes/<node>/tasks --limit 50 --output-filter status:running)?
3. Is the Dev VPS resource pool exhausted? (No available host/memory/disk in the pool — orders queue but never assign.)
4. Is the module-group-7 → Proxmox bridge healthy? Orders reach Blesta but the handoff to the hypervisor layer appears to silently fail (no error surfaced back through the API).

Acceptance criteria (what closes this issue)

• python3 scripts/dev-zone-probe.py exits 0 (VM reaches service_status=active with an IP within 300s)
• python3 scripts/dev-zone-probe.py --template debian13-cloud also exits 0 (so VM cloud-init deadlock on debian13-cloud template — SSH never starts #24 can then be verified)
• Provisioning time is within ~2× the Katy TX baseline (~100s → accept up to ~200s for Cherryvale)

Workaround in the meantime

• Use the NVMe zone (Katy, TX) for any VM workload that doesn't require nested KVM. Package 23 ($0.26/day) is the smallest and provisions reliably.
• CI: pin integration tests to the NVMe zone until this resolves (tracked in terraform-provider-shc#13).
• Nested-KVM workloads (Firecracker, QEMU-in-VM, the ephemeral GitHub-runner fleet) are blocked until Cherryvale is back — there is no nested-KVM zone other than Dev VPS.

[Amperstrand]

Isolation Test Confirms: #24 is the Same Issue

Tested debian13-cloud in Katy-TX → works perfectly (53s, SSH reachable). The cloud-init deadlock in #24 only happens in Cherryvale-KS. Both issues have the same root cause: broken Cherryvale-KS infrastructure.

Consolidating #24 into #28.