Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/content/install/platform-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,46 @@ Check that LWS deployments are ready:

## Install Gateway API Controller

!!! warning "Do not install OpenShift Service Mesh manually (OLM-managed OCP versions)"
On OCP versions where the `openshift-ingress` ClusterOperator uses OLM to
manage OSSM 3, manually installing OSSM can leave the GatewayClass in
`Accepted: Unknown` ("Waiting for controller") and block all Gateway API
traffic — including `maas-default-gateway`.

**Which versions are affected?** The ingress operator is migrating from OLM
to the Sail Library for OSSM management. The conflict described here applies
only to OCP versions still on the OLM path:

| OCP version | OSSM management | Manual OSSM conflict? |
|-------------|-----------------|----------------------|
| 4.19 | OLM | **Yes** (backport TBD) |
| 4.20 | OLM | **Yes** (direct-install backport in progress) |
| 4.21 < 4.21.22 | OLM | **Yes** |
| 4.21.22+ | Sail Library | No — CIO installs Istio directly |
| 4.22+ | Sail Library | No — shipped without OLM from day one |

On OLM-managed versions, two common conflicts occur:

- **OSSM v2.x subscription present:** OSSM v2 and v3 cannot coexist. An active
`servicemeshoperator` (v2) subscription prevents the ingress operator from
installing OSSM v3, and the `clusteroperator/ingress` reports
`GatewayAPIOSSMConflict`.
- **OSSM v3 installed manually:** Manually installing `servicemeshoperator3`
(e.g. from the OperatorHub stable channel) pins a version that may differ from
what the ingress operator expects, preventing it from managing the OSSM
lifecycle.

Additionally, any operator installed into the `openshift-operators` namespace can
pull an OSSM upgrade into its InstallPlan (OLM v0 limitation), re-breaking the
gateway even after the conflicting subscription is removed.

**If you have a conflicting OSSM subscription:** delete the
`servicemeshoperator` (v2) or `servicemeshoperator3` (v3) Subscription and CSV.
The `openshift-ingress` operator will install OSSM at the correct pinned version
automatically. Do **not** approve the upgrade InstallPlan that appears afterward.
See [Troubleshooting — GatewayClass stuck](troubleshooting.md#common-issues)
(issue 12) for step-by-step remediation.

Initialize OpenShift's provided Gateway API implementation:

```yaml
Expand Down
47 changes: 43 additions & 4 deletions docs/content/install/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,48 @@ This guide helps you diagnose and resolve common issues with MaaS Platform deplo

See [OdhDashboardConfig Feature Flags](maas-setup.md#odhdashboardconfig-feature-flags) for setup.

12. **TLS certificate errors (`curl: (60) SSL certificate problem`)**: Your cluster uses self-signed or internal CA certificates that are not in your system trust store. See [TLS Certificate Validation](#tls-certificate-validation) below.
12. **GatewayClass stuck in `Accepted: Unknown` ("Waiting for controller")**: A conflicting OSSM subscription prevents the `openshift-ingress` operator from managing Gateway API on OCP versions where the ingress operator uses OLM for OSSM management (4.19, 4.20, 4.21 before 4.21.22). On OCP 4.21.22+, 4.22+, the ingress operator uses the Sail Library directly and manual OSSM subscriptions do not cause this conflict.

13. **Cannot create MaaSSubscription or MaaSAuthPolicy (`no endpoints available for service "maas-controller-webhook-service"`)**: The maas-controller pods are not running or not ready.
On affected versions, the `openshift-ingress` ClusterOperator manages OSSM 3 via OLM and
pins it to a version compatible with the cluster. Two scenarios cause this failure:

- An **OSSM v2.x** subscription (`servicemeshoperator`) blocks OSSM v3 installation —
v2 and v3 cannot coexist, and the ingress operator reports `GatewayAPIOSSMConflict`.
- A **manually installed OSSM v3** subscription (`servicemeshoperator3`, e.g. from the
stable channel via OperatorHub) pins a version the ingress operator cannot manage.

- [ ] Check for a conflicting OSSM subscription (v2 or v3):

```bash
kubectl get subscription -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' | grep -i servicemesh
```

- [ ] Delete the conflicting subscription and its CSV (replace `<namespace>` and `<operator-name>` with the values from the previous step, e.g. `openshift-operators` and `servicemeshoperator3` or `servicemeshoperator`):

```bash
kubectl delete subscription <operator-name> -n <namespace>
kubectl delete csv -n <namespace> -l operators.coreos.com/<operator-name>.<namespace>
```

- [ ] Wait for the `openshift-ingress` operator to reinstall OSSM at the pinned version:

```bash
kubectl wait --for=condition=Available clusteroperator/ingress --timeout=300s
```

- [ ] Do **not** approve the OSSM upgrade InstallPlan that may appear in `openshift-operators` — approving it re-breaks the gateway

- [ ] Verify the GatewayClass is now accepted (`Accepted` must be `True`):

```bash
kubectl get gatewayclass openshift-default -o jsonpath='{.metadata.name}{"\t"}{range .status.conditions[?(@.type=="Accepted")]}{.status}{"\t"}{.message}{end}{"\n"}'
```
Comment thread
jira-autofix[bot] marked this conversation as resolved.

See [Install Gateway API Controller](platform-setup.md#install-gateway-api-controller) for the full warning and context.

13. **TLS certificate errors (`curl: (60) SSL certificate problem`)**: Your cluster uses self-signed or internal CA certificates that are not in your system trust store. See [TLS Certificate Validation](#tls-certificate-validation) below.

14. **Cannot create MaaSSubscription or MaaSAuthPolicy (`no endpoints available for service "maas-controller-webhook-service"`)**: The maas-controller pods are not running or not ready.

MaaS uses admission webhooks to validate resource creation. When the controller is unavailable (pod crash, upgrade, or scaled to 0), the webhook endpoint becomes unreachable and creates are rejected.

Expand All @@ -124,7 +163,7 @@ This guide helps you diagnose and resolve common issues with MaaS Platform deplo

Creates succeed once controller pods are healthy. Model inference requests are unaffected during controller downtime (data plane continues operating normally).

14. **Cannot create `AITenant` (`must be created in the configured AITenant infrastructure namespace`)**: The object is being created outside the namespace configured by `--aitenant-namespace` (default `ai-tenants`).
15. **Cannot create `AITenant` (`must be created in the configured AITenant infrastructure namespace`)**: The object is being created outside the namespace configured by `--aitenant-namespace` (default `ai-tenants`).
Comment thread
jira-autofix[bot] marked this conversation as resolved.

- [ ] Check which namespace the controller is configured to accept:

Expand All @@ -138,7 +177,7 @@ This guide helps you diagnose and resolve common issues with MaaS Platform deplo
kubectl get namespace ai-tenants
```

- [ ] If the error is `no endpoints available for service "maas-controller-webhook-service"`, follow the same webhook health checks as issue 12 above.
- [ ] If the error is `no endpoints available for service "maas-controller-webhook-service"`, follow the same webhook health checks as issue 14 above.

## Conflicting AuthPolicy Detection

Expand Down
2 changes: 1 addition & 1 deletion scripts/setup-gateway.sh
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ setup_route_mode() {
# the Route can be created in the next step (see wait_gateway_programmed).
log_info " Waiting for Gateway to be Programmed (timeout: ${GATEWAY_TIMEOUT}s)..."
if ! kubectl wait --for=condition=Programmed gateway/"$GATEWAY_NAME" -n "$GATEWAY_NAMESPACE" --timeout="${GATEWAY_TIMEOUT}s" 2>/dev/null; then
log_warn "Gateway not Programmed after ${GATEWAY_TIMEOUT}s - check Service Mesh installation"
log_warn "Gateway not Programmed after ${GATEWAY_TIMEOUT}s - on OCP versions where the ingress operator uses OLM for OSSM (4.19, 4.20, 4.21 before 4.21.22), check for a conflicting OSSM subscription (servicemeshoperator v2 or manually installed servicemeshoperator3). On 4.21.22+ / 4.22+ the ingress operator uses the Sail Library and manual OSSM subscriptions do not conflict. Run: kubectl get subscription -A | grep servicemesh"
else
log_info " Gateway is Programmed"
fi
Expand Down
Loading