Conversation
- Add url field to inference secret template (values-secret.yaml.template,
inference-secret.yaml ExternalSecret) so a custom base URL can be stored
in Vault and propagated via ESO
- Extend setup-bom-profiles.sh to read /ws-secrets/{secret}/url and write
PROV_{NAME}_URL into bom.env, carrying the URL into the gateway VM
- Add Provider.url field and "custom" entry to PROVIDER_CRED_MAP in
apply_bom.py; parse_profiles() resolves url from YAML or PROV_{NAME}_URL
env; create_provider() passes --config base_url=<url> when set;
onboard_nemoclaw() sets NEMOCLAW_INFERENCE_BASE_URL; start_openclaw_gateway()
accepts provider_base_url parameter instead of hardcoded URL
- Add governance-policy chart customEndpointHost value and conditional
"custom" profile rendering in the governance-interceptor-profiles ConfigMap
(required: interceptor blocks provider creation for unregistered types)
- Add data-science/custom BOM profile directory for the custom provider type
- Add 8 unit tests covering url field defaults, YAML/env resolution, priority,
create_provider --config injection, and onboard_nemoclaw env propagation
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Wire the custom vLLM governance profile to the TinyLlama deployment route on the test cluster. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The data-science profile activates all workspace subdirectories together. Using 'default' as the workspace name caused a sandbox collision with the existing default/notebook sandbox. Using 'vllm' gives the custom endpoint its own workspace without conflict. A follow-up ticket (local-docs/jira-profile-selection.md) tracks making profile selection provider-aware so users only get the workspace they need. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
curl rejects HTTP/0.9 responses (new default in curl v8), and the interceptor speaks gRPC which sends HTTP/0.9-style framing. On a fresh install the openshell-gateway service isn't running yet either, so the journalctl fallback also fails. Replace both with a bash /dev/tcp check: if the port accepts connections, the interceptor is up. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Without this, setup-bom-profiles.sh wrote PROV_{name}_URL for every
provider sharing the inference secret, causing the vLLM base URL to
leak onto unrelated providers (e.g. nvidia) that happened to use the
same credentialSecret.
- Add urlSecretKey field to providers.yaml spec; setup-bom-profiles.sh
only writes PROV_{name}_URL when urlSecretKey is declared
- parse_profiles() in apply_bom.py only reads PROV_{name}_URL from env
when the provider declares urlSecretKey (mirrors the shell behaviour)
- Update custom/providers.yaml to declare urlSecretKey: url
- Add test_parse_profiles_no_url_from_env_without_urlsecretkey to guard
the regression
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…lock The patterns operator tries to delete child apps through an ACM hub, which doesn't exist in single-cluster deployments. This causes the Pattern CR finalizer to loop indefinitely with "waiting N hub child applications to be removed". Fix: pre-uninstall now removes ArgoCD app finalizers and deletes the applications before rhvp.cluster_utils.uninstall triggers Pattern CR deletion. With no apps left, the operator's finalizer completes immediately. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…tion deadlock When external-secrets-operator is uninstalled before the openshell-agents namespace finishes terminating, the ESO ValidatingWebhookConfiguration intercepts ExternalSecret deletions and fails (webhook service gone), leaving the namespace stuck in Terminating indefinitely. Pre-uninstall now deletes both ESO webhook configurations before the namespace cleanup begins. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
CNV 4.22.9 changed the CRD schema for spec.featureGates from object to
array. The chart had spec: {} which left the previous object value in
place via strategic merge, causing OLM InstallPlan validation to fail
on every reinstall.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…y webhook The CNV admission webhook rejects featureGates: [] (array format). The OLM InstallPlan schema conflict was caused by a stale CR from a previous install, not the spec value. The correct fix is to delete the old CR before upgrade (handled by pre-uninstall) rather than changing the spec format. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
When the HyperConverged CR from a previous install is still present during OLM InstallPlan execution, the new CRD schema validation fails with a type conflict (featureGates object vs array between v1beta1 and v1 CRD versions). This causes a reinstall deadlock. pre-uninstall now: removes HyperConverged finalizers so deletion isn't blocked, then waits up to 60s for the CR to be fully gone before proceeding. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The App-of-Apps was recreating child apps (openshell-saw, openshift-cnv) between their deletion and the Pattern finalizer check, keeping the deadlock alive. Also wait 30s for deletion to complete so apps are fully gone before the Pattern CR finalizer runs. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The ODH gateway binary refreshes its supervisor from the upstream Docker image (ghcr.io/nvidia/openshell/supervisor:dev) at startup, expecting the binary at /openshell-sandbox. When that path is absent in the upstream :dev tag, the gateway crashes before listening on port 17670, making all subsequent BOM openshell commands fail with Connection refused. After extracting the supervisor binary from the ODH image, pre-populate the gateway's content-addressed cache so the runtime refresh is a cache hit and the gateway starts successfully. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Based on live cluster testing: - The governance interceptor maps --type to a profile ID (custom → custom profile, openai → openai profile). The openai type is the correct choice for vLLM since it exposes an OpenAI-compatible API. - Rename governance profile from custom.yaml → openai.yaml with provider_type: openai, inference_capable: false, and explicit binaries allowlist (node + curl) so the governance CONNECT proxy permits direct sandbox egress to the vLLM endpoint. - BOM custom profile: rename provider from custom → openai type. - PROVIDER_CRED_MAP: custom → openai entry. - Override file pattern: governance-policy.yaml in overrides/ (mirrors openshell-saw.yaml) so customEndpointHost reaches the chart via extraValueFiles instead of the broken helmOverrides mechanism. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ream image The upstream ghcr.io/nvidia/openshell/supervisor:dev image ships the binary at /openshell-supervisor, but the gateway looks for /openshell-sandbox in the content-addressed cache. Previous fix used the wrong source path. Also pull the upstream image fresh before inspecting the digest, so the cache entry matches the digest the gateway will see on startup. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The ESO operator recreates the ValidatingWebhookConfiguration between pre-uninstall (which removes it) and the namespace deletion, blocking namespace termination indefinitely. Post-uninstall: remove the webhook again and force-finalize the namespace via the Kubernetes API proxy if it's still Terminating after the ansible uninstall completes. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Multi-line heredoc inside guest_ssh caused silent failures due to quoting/continuation issues. Collapse to a single semicolon-separated line which works correctly through virtctl ssh --command. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The provider was renamed from 'custom' to 'openai' in providers.yaml but sandbox.yaml still referenced 'custom', causing sandbox creation to fail with 'custom is not a recognized provider type'. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…cache The upstream ghcr.io/nvidia/openshell/supervisor:dev image ships two binaries: /openshell-supervisor (host-side lifecycle manager) and the actual in-container sandbox supervisor. Previous fix extracted the wrong one (/openshell-supervisor), causing sandboxes to crash immediately with '--backend-descriptor-file is required for --role=isolation-backend'. Always copy /usr/local/bin/openshell-supervisor (extracted from the ODH image's /openshell-sandbox during the supervisor upgrade step) into the gateway's content-addressed cache. The chmod ensures the file is writable if the gateway already wrote a wrong version there. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Without the execute bit, the sandbox container fails to start with 'permission denied' on /opt/openshell/bin/openshell-sandbox. The chmod 644 from the first pass was overriding the execute bit. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
30 minutes is insufficient when multiple BOM workspaces have provider type mismatches causing 20x5s polling loops, plus the upgrade phase. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
When inference.provider=custom, the default workspace's nvidia provider is intentionally not created (type mismatch). The verifier was failing on these missing providers, causing the setup job to exit 1 and restart indefinitely. Verifier now calls check_provider_type_mismatch() before checking each provider. Mismatched providers are logged as SKIP, and sandboxes whose only providers were all skipped are also skipped. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The gateway crashes on startup when the upstream image digest changes between the upgrade pre-population step and the gateway's own docker pull at startup. Adding an ExecStartPre drop-in ensures the ODH supervisor binary is always in the content-addressed cache immediately before the gateway starts, regardless of which image digest was pulled. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Suggested in PR review on validatedpatterns-sandbox#45: without a start limit, Restart=always retries forever if the sandbox is deleted, which can exhaust system resources. Cap at 10 restarts within a 5-minute window — enough to recover from transient drops, not enough to spin forever. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Hadar301
changed the title
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Covers: getting an OpenAI-compatible endpoint, configuring the inference secret and governance override, installing, verifying end-to-end, and how the governance proxy enforces sandbox egress. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The gateway does its own docker pull at startup which may get a newer digest than what ExecStartPre inspected. By pulling in ExecStartPre first, both use the same digest and the cache hit is guaranteed. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
When inference.provider=custom, PROV_OPENAI_TYPE=custom is set from the secret. check_provider_type_mismatch() checks if the configured type matches provider.type OR provider.nemoclaw_provider. Setting nemoclawProvider: custom makes 'custom' a valid match, so the openai provider is created instead of being silently skipped. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ecreation The App-of-Apps (secure-agent-workspace-prod) was recreating child apps (openshell-saw, openshift-cnv) between our bulk delete and the Pattern CR finalizer check. Delete the App-of-Apps first and wait for it to be gone so it can't regenerate children, then delete remaining child apps. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ystemd unit
Systemd converts {{ to { in unit file content, making {{.Id}} → {.Id}
which is invalid Go template syntax. Docker then returns empty string
and the cache pre-population never happens.
docker images --no-trunc -q returns the full sha256 digest directly,
with no Go template needed — safe in systemd unit files.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Inside double-quoted guest_ssh "...", bash expands $D and $(...) before sending to SSH. Escaping with \$ preserves them so the VM receives the literal dollar signs needed for the ExecStartPre command. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Replace nested SSH/systemd commands with a cache helper that avoids Helm template conflicts. Read the custom model from the inference secret instead of falling back to the NVIDIA model. Add regression tests and update the custom provider documentation.
- preserve compatibility with inference secrets without a URL - bound uninstall and propagate provisioning and dashboard failures - skip incompatible sandboxes and scope lookups by workspace - add regression tests and update deployment documentation
- reconcile network MTU and add a persistent TCP MSS safeguard - add regression tests and document inference timeout troubleshooting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a custom OpenAI-compatible inference endpoint to the secrets pipeline, governance policy, and BOM provisioning. The
vllm/notebooksandbox is created automatically with the configured endpoint and model, includingtinyllama:latest.The provider path was validated on OpenShift. Follow-up readiness fixes are currently in the local working tree and must be committed and pushed before this PR is considered merge-ready.
Changes
url; custom providers require a nonempty URL and model.customEndpointHostin the governance override to the HTTPS route hostname before installation.UNINSTALL_TIMEOUT_SECONDS(default 600), preserve playbook errors, terminate launched processes on timeout, and stop subsequent forced cleanup.Configuration and validation
Set inference
provider: custom,model: <served-model-id>,api_key: <endpoint-key>, andurl: https://<inference-route-hostname>/v1. Set governancecustomEndpointHostto the same hostname. See the custom inference guide for complete examples and separate checks for sandbox readiness, OpenClaw readiness, dashboard access, and actual inference.Live deployment of d557e39: supervisor-cache hook succeeded, gateway stayed running, OIDC succeeded, and
vllm/notebookbecame Ready. OpenClaw selectedtinyllama:latestand passed its readiness check. BOM verification reported 7 passed and 0 failed, with NVIDIA-dependent sandboxes skipped. Dashboard setup failed with a unit-file permission error, motivating the local follow-up fix.Local follow-up validation: 57 regression tests passed, covering for optional URL templating, missing required fields, provider skips, sandbox failures/timeouts, workspace-specific lookup, dashboard file/symlink replacement and failures, and uninstall termination. Helm rendering, shell syntax, and Make dry-run checks are included. The follow-up changes have not been rolled out; dashboard login and an actual inference response still need live verification.
Remaining limitations
APPENG-6328