Skip to content

Fix Triton failing to start: move its HTTP port off Ray Serve's 8000 #218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kondratyevd merged 1 commit into from
Sep 7, 2026
Merged

Fix Triton failing to start: move its HTTP port off Ray Serve's 8000 #218

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 39 additions & 19 deletions apps/ray/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ No custom image, no protocol code, no model code: official Ray, official
Triton, ~100 lines of glue shipped as a ConfigMap, Triton's Python stubs
pip-installed by an init container.

| path | what it is |
| --- | --- |
| `helmrepo.yaml` | `HelmRepository` for the KubeRay charts |
| `operator/` | `kuberay-operator` 1.7.0 — the `ray.io` CRDs and the controller. Namespaced (`singleNamespaceInstall: true`), so both the watch and the RBAC stay in `cms`. |
| `sonic-ray/chart/` | the `sonic-ray` chart: a `RayService` with a Triton in every worker pod and the forwarder as its Serve application, the ConfigMap carrying the forwarder, two metrics Services |
| `sonic-ray/chart/files/sonic_ray/serve_app.py` | the forwarder — one replica per pod, every unary RPC of `GRPCInferenceService` handed to the pod's Triton unchanged |
| `sonic-ray/helmrelease.yaml`, `sonic-ray/values.yaml` | the AF release: `dependsOn` the operator, values with the `triton:` block of the `supersonic` release's values |
| [`tests/sonic_ray/`](../../tests/sonic_ray), [`tests/manifests/test_ray.py`](../../tests/manifests/test_ray.py) | source-level checks of the forwarder; rendered-chart checks incl. parity with `apps/sonic/supersonic/values.yaml` |
| path | what it is |
| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `helmrepo.yaml` | `HelmRepository` for the KubeRay charts |
| `operator/` | `kuberay-operator` 1.7.0 — the `ray.io` CRDs and the controller. Namespaced (`singleNamespaceInstall: true`), so both the watch and the RBAC stay in `cms`. |
| `sonic-ray/chart/` | the `sonic-ray` chart: a `RayService` with a Triton in every worker pod and the forwarder as its Serve application, the ConfigMap carrying the forwarder, two metrics Services |
| `sonic-ray/chart/files/sonic_ray/serve_app.py` | the forwarder — one replica per pod, every unary RPC of `GRPCInferenceService` handed to the pod's Triton unchanged |
| `sonic-ray/helmrelease.yaml`, `sonic-ray/values.yaml` | the AF release: `dependsOn` the operator, values with the `triton:` block of the `supersonic` release's values |
| [`tests/sonic_ray/`](../../tests/sonic_ray), [`tests/manifests/test_ray.py`](../../tests/manifests/test_ray.py) | source-level checks of the forwarder; rendered-chart checks incl. parity with `apps/sonic/supersonic/values.yaml` |

The chart lives here (like `apps/sonic/model-manager`) rather than being a raw
`RayService` because of ordering: until the operator's chart has installed the
Expand Down Expand Up @@ -57,6 +57,26 @@ else. Nothing else claims the resource, which is what leaves a pod without a
replica idle and therefore reclaimable, and a replica without a pod pending —
the request that grows the group.

### Ports

Both containers share the pod's network namespace, so they cannot both use
Triton's defaults:

| port | who | why it is where it is |
| ---- | -------------------- | -------------------------------------------------------------------------------------- |
| 8000 | Ray Serve HTTP proxy | KubeRay probes the proxy here; it does not move |
| 9000 | Ray Serve gRPC proxy | the inference entry point, behind the release's Service on 8001 |
| 8080 | Ray metrics | scraped by `sonic-ray-metrics` |
| 8100 | Triton HTTP | **moved** off Triton's default 8000, which Ray holds; serves only the kubelet's probes |
| 8001 | Triton gRPC | Triton's default; what the forwarder dials on localhost |
| 8002 | Triton metrics | scraped by `sonic-ray-triton-metrics` |

Leaving Triton on 8000 is what makes it exit with `failed to start HTTP
service: Unavailable - Socket '0.0.0.0:8000' already in use`. The ports are
chart values (`triton.httpPort`, `triton.grpcPort`), and the chart refuses to
render if they collide with Ray's, if they disagree with the `--http-port` /
`--grpc-port` in `triton.args`, or if the args leave Triton on 8000.

## What it serves and speaks

Whatever Triton is pointed at. On the AF that is the models CMSSW ships:
Expand Down Expand Up @@ -108,7 +128,7 @@ Two loops, both Ray's, nothing else in between:
`terminationGracePeriodSeconds` against Triton's `--exit-timeout-secs` to
drain (the chart refuses to render if the first is not larger).

One pair of numbers sizes both, because a replica *is* a pod. Raising the GPU
One pair of numbers sizes both, because a replica _is_ a pod. Raising the GPU
ceiling is one edit in `sonic-ray/values.yaml`:

```yaml
Expand All @@ -122,16 +142,16 @@ reclaims a pod that stays broken.

## How it lines up with SuperSONIC

| SuperSONIC (`supersonic`) | Ray (`sonic-ray`) |
| --- | --- |
| Triton on a per-site PVC or CVMFS, explicit load list | Triton on CVMFS, explicit load list — a plain `--model-repository` path |
| Envoy: gRPC entry point behind a `LoadBalancer` on `geddes-private-pool`, `ROUND_ROBIN` | Serve's gRPC proxy behind KubeRay's serve Service, same pool, port 8001 |
| `ingress.enabled: false` — private pool only | no ingress; the head is `ClusterIP`, dashboard by port-forward only |
| KEDA `ScaledObject` on a Prometheus expression, 1–10 pods | Ray Serve request-based autoscaling, 1–4 pods — see above |
| `nodeSelector: cms-af-prod=true` + the `hub.jupyter.org/dedicated` toleration | same, head and workers |
| model repository from a PVC or CVMFS | the cluster's `cvmfs` claim, mounted **read-only** |
| Triton Service labelled `scrape_metrics: "true"` | `sonic-ray-triton-metrics` (`nv_*`) and `sonic-ray-metrics` (Ray, incl. `ray_serve_*`), same label, `release="sonic-ray"` |
| Envoy's Lua rate limiter on `RepositoryIndex` | none; Serve's `maxOngoingRequests` back-pressure instead |
| SuperSONIC (`supersonic`) | Ray (`sonic-ray`) |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Triton on a per-site PVC or CVMFS, explicit load list | Triton on CVMFS, explicit load list — a plain `--model-repository` path |
| Envoy: gRPC entry point behind a `LoadBalancer` on `geddes-private-pool`, `ROUND_ROBIN` | Serve's gRPC proxy behind KubeRay's serve Service, same pool, port 8001 |
| `ingress.enabled: false` — private pool only | no ingress; the head is `ClusterIP`, dashboard by port-forward only |
| KEDA `ScaledObject` on a Prometheus expression, 1–10 pods | Ray Serve request-based autoscaling, 1–4 pods — see above |
| `nodeSelector: cms-af-prod=true` + the `hub.jupyter.org/dedicated` toleration | same, head and workers |
| model repository from a PVC or CVMFS | the cluster's `cvmfs` claim, mounted **read-only** |
| Triton Service labelled `scrape_metrics: "true"` | `sonic-ray-triton-metrics` (`nv_*`) and `sonic-ray-metrics` (Ray, incl. `ray_serve_*`), same label, `release="sonic-ray"` |
| Envoy's Lua rate limiter on `RepositoryIndex` | none; Serve's `maxOngoingRequests` back-pressure instead |

## Using it

Expand Down
33 changes: 33 additions & 0 deletions apps/ray/sonic-ray/chart/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ mounts, and the init container that pip-installs Triton's stubs.
{{- define "sonic-ray.rayEnv" -}}
- name: PYTHONPATH
value: {{ printf "%s:%s" (include "sonic-ray.codeDir" .) (include "sonic-ray.depsDir" .) | quote }}
# Where the forwarder dials the Triton in its own pod.
- name: TRITON_GRPC
value: {{ printf "localhost:%d" (int .Values.triton.grpcPort) | quote }}
{{- end -}}

{{- define "sonic-ray.rayMounts" -}}
Expand Down Expand Up @@ -115,6 +118,36 @@ Refuse to render what cannot work.
{{- fail "triton.resources.limits must request exactly one nvidia.com/gpu: a pod is one Triton on one GPU." -}}
{{- end -}}
{{- $args := join " " .Values.triton.args -}}
{{/*
Triton and Ray share the pod's network namespace. Ray binds 8000 (Serve HTTP
proxy — KubeRay probes it there, so it is not the one that moves), 9000
(Serve gRPC proxy) and 8080 (metrics); Triton must sit elsewhere and its args
must say the same numbers as the values, or the probes and the forwarder
would address a port nothing listens on.
*/}}
{{- range $name, $port := dict "httpPort" .Values.triton.httpPort "grpcPort" .Values.triton.grpcPort -}}
{{- if has (int $port) (list 8000 8080 9000) -}}
{{- fail (printf "triton.%s is %d, which Ray binds in this pod (8000 Serve HTTP, 9000 Serve gRPC, 8080 metrics). Triton would fail to bind it." $name (int $port)) -}}
{{- end -}}
{{- end -}}
{{- if eq (int .Values.triton.httpPort) (int .Values.triton.grpcPort) -}}
{{- fail "triton.httpPort and triton.grpcPort are the same port." -}}
{{- end -}}
{{- $http := regexFind "--http-port=[0-9]+" $args -}}
{{- if not $http -}}
{{- fail (printf "triton.args must pass --http-port=%d: Triton defaults to 8000, which Ray Serve's HTTP proxy already binds in this pod." (int .Values.triton.httpPort)) -}}
{{- end -}}
{{- if ne (trimPrefix "--http-port=" $http | int) (int .Values.triton.httpPort) -}}
{{- fail (printf "triton.args say %s but triton.httpPort is %d: the probes would address a port Triton does not listen on." $http (int .Values.triton.httpPort)) -}}
{{- end -}}
{{- $grpc := regexFind "--grpc-port=[0-9]+" $args -}}
{{- if $grpc -}}
{{- if ne (trimPrefix "--grpc-port=" $grpc | int) (int .Values.triton.grpcPort) -}}
{{- fail (printf "triton.args say %s but triton.grpcPort is %d: the forwarder would dial a port Triton does not listen on." $grpc (int .Values.triton.grpcPort)) -}}
{{- end -}}
{{- else if ne (int .Values.triton.grpcPort) 8001 -}}
{{- fail (printf "triton.grpcPort is %d but triton.args do not pass --grpc-port, so Triton listens on its default 8001." (int .Values.triton.grpcPort)) -}}
{{- end -}}
{{- if not (contains .Values.triton.modelRepository.mountPath $args) -}}
{{- fail (printf "triton.args never mention triton.modelRepository.mountPath (%s): Triton would not see the repository that is mounted." .Values.triton.modelRepository.mountPath) -}}
{{- end -}}
Expand Down
6 changes: 4 additions & 2 deletions apps/ray/sonic-ray/chart/templates/rayservice.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,10 @@ spec:
args:
{{- toYaml .Values.triton.args | nindent 18 }}
ports:
- { name: http, containerPort: 8000 }
- { name: grpc, containerPort: 8001 }
# Named, because the probes address http by name. Neither is
# Triton's default: Ray has 8000 in this pod (see values).
- { name: http, containerPort: {{ .Values.triton.httpPort }} }
- { name: grpc, containerPort: {{ .Values.triton.grpcPort }} }
- { name: triton-metrics, containerPort: 8002 }
resources:
{{- toYaml .Values.triton.resources | nindent 18 }}
Expand Down
17 changes: 15 additions & 2 deletions apps/ray/sonic-ray/chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,25 @@ triton:
repository: nvcr.io/nvidia/tritonserver
tag: "26.04-py3"
pullPolicy: IfNotPresent
# -- Passed to the Triton container as-is; the model repository path in args
# must be modelRepository.mountPath.
# -- Passed to the Triton container as-is. The model repository path in args
# must be modelRepository.mountPath, and the ports in args must be the ones
# below — the chart refuses to render otherwise.
command: ["tritonserver"]
args:
- --model-repository=/models
- --http-port=8100
- --exit-timeout-secs=60
# -- Ports Triton listens on. The Ray container shares this pod's network
# namespace, so these must avoid the ports Ray binds there: 8000 (Serve's
# HTTP proxy), 9000 (Serve's gRPC proxy), 8080 (metrics). HTTP therefore
# cannot be Triton's default 8000 — it would fail to bind with "Socket
# '0.0.0.0:8000' already in use" — and Serve's proxy is the one that has to
# keep 8000, since KubeRay probes the proxy there. Nothing outside the pod
# reaches either of these: inference arrives at Serve's gRPC proxy, which
# forwards to grpcPort on localhost, and httpPort serves only the kubelet's
# probes.
httpPort: 8100
grpcPort: 8001
# -- One GPU per Triton is the unit of scaling: a pod is one Triton, and a
# Serve replica is one pod.
resources:
Expand Down
6 changes: 6 additions & 0 deletions apps/ray/sonic-ray/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ triton:
args:
- |
/opt/tritonserver/bin/tritonserver \
--http-port=8100 \
--model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoBTag/Combined/data/models/ \
--model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoEgamma/EgammaPhotonProducers/data/models/ \
--model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoTauTag/TrainingFiles/data/DeepTauIdSONIC/ \
Expand Down Expand Up @@ -55,6 +56,11 @@ triton:
failureThreshold: 10
successThreshold: 1
httpGet: { path: /v2/health/ready, port: http }
# Triton's HTTP moves off its default 8000: the Ray container in this pod
# binds that for Ray Serve's HTTP proxy, and containers of a pod share one
# network namespace. gRPC keeps 8001, which is what the forwarder dials.
httpPort: 8100
grpcPort: 8001
# The cluster's CVMFS claim (apps/infrastructure/cvmfs-pvc.yaml), read-only.
# HostToContainer: CVMFS repositories are autofs mounts on the node, and the
# pod must see the ones mounted after it started.
Expand Down
48 changes: 46 additions & 2 deletions tests/manifests/test_ray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,12 +271,53 @@ def test_rendered_triton_is_the_one_in_values(worker_group, values):
assert triton["readinessProbe"] == values["triton"]["readinessProbe"]
assert triton["startupProbe"] == values["triton"]["startupProbe"]
assert {p["name"]: p["containerPort"] for p in triton["ports"]} == {
"http": 8000,
"grpc": 8001,
"http": values["triton"]["httpPort"],
"grpc": values["triton"]["grpcPort"],
"triton-metrics": 8002,
}


def test_triton_and_ray_do_not_fight_over_a_port(worker_group, values):
"""Containers of a pod share one network namespace. Triton's defaults
collide with Ray Serve's HTTP proxy on 8000 — it fails to start with
"Socket '0.0.0.0:8000' already in use" — so Triton moves, and Serve keeps
8000 because KubeRay probes the proxy there. The args must name the same
ports as the values, or the probes and the forwarder address nothing."""
declared = [
(c["name"], p["name"], p["containerPort"])
for c in worker_group["template"]["spec"]["containers"]
for p in c.get("ports", [])
]
numbers = [port for _, _, port in declared]
assert len(numbers) == len(set(numbers)), (
f"two containers claim one port: {declared}"
)

ray_ports = {port for owner, _, port in declared if owner == "ray-worker"}
assert 8000 in ray_ports, (
"Serve's HTTP proxy must keep 8000: KubeRay probes it there"
)
triton = values["triton"]
assert triton["httpPort"] != 8000
args = triton["args"][0].split()
assert f"--http-port={triton['httpPort']}" in args
# gRPC is left at Triton's default, which is what the forwarder dials.
assert triton["grpcPort"] == 8001
assert not [a for a in args if a.startswith("--grpc-port=")]


def test_the_forwarder_is_told_where_its_triton_is(head_pod, worker_group, values):
"""The forwarder dials TRITON_GRPC; unset, it falls back to a default that
would be wrong the moment grpcPort changed."""
expected = f"localhost:{values['triton']['grpcPort']}"
for template, name in (
(head_pod, "ray-head"),
(worker_group["template"], "ray-worker"),
):
assert env_of(container(template, name))["TRITON_GRPC"] == expected
assert f'"TRITON_GRPC", "{expected}"' in SERVE_APP.read_text()


def test_models_come_from_cvmfs_read_only(worker_group, values):
"""The cluster's CVMFS claim, mounted read-only with host-to-container
propagation (CVMFS repositories are autofs mounts on the node), and every
Expand Down Expand Up @@ -501,6 +542,9 @@ def test_metrics_services_select_labels_kuberay_leaves_alone(
[
("triton.modelRepository.claimName=", "claimName is required"),
("replicas.min=5", "replicas.min exceeds replicas.max"),
("triton.httpPort=8000", "which Ray binds in this pod"),
("triton.httpPort=8200", "but triton.httpPort is 8200"),
("triton.grpcPort=8500", "do not pass --grpc-port"),
("triton.resources.limits.nvidia\\.com/gpu=2", "exactly one nvidia.com/gpu"),
(
"ray.worker.terminationGracePeriodSeconds=30",
Expand Down