diff --git a/apps/ray/README.md b/apps/ray/README.md index c75f2d8b..8133d85f 100644 --- a/apps/ray/README.md +++ b/apps/ray/README.md @@ -13,14 +13,14 @@ No custom image, no protocol code, no model code: official Ray, official Triton, ~100 lines of glue shipped as a ConfigMap, Triton's Python stubs pip-installed by an init container. -| path | what it is | -| --- | --- | -| `helmrepo.yaml` | `HelmRepository` for the KubeRay charts | -| `operator/` | `kuberay-operator` 1.7.0 — the `ray.io` CRDs and the controller. Namespaced (`singleNamespaceInstall: true`), so both the watch and the RBAC stay in `cms`. | -| `sonic-ray/chart/` | the `sonic-ray` chart: a `RayService` with a Triton in every worker pod and the forwarder as its Serve application, the ConfigMap carrying the forwarder, two metrics Services | -| `sonic-ray/chart/files/sonic_ray/serve_app.py` | the forwarder — one replica per pod, every unary RPC of `GRPCInferenceService` handed to the pod's Triton unchanged | -| `sonic-ray/helmrelease.yaml`, `sonic-ray/values.yaml` | the AF release: `dependsOn` the operator, values with the `triton:` block of the `supersonic` release's values | -| [`tests/sonic_ray/`](../../tests/sonic_ray), [`tests/manifests/test_ray.py`](../../tests/manifests/test_ray.py) | source-level checks of the forwarder; rendered-chart checks incl. parity with `apps/sonic/supersonic/values.yaml` | +| path | what it is | +| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `helmrepo.yaml` | `HelmRepository` for the KubeRay charts | +| `operator/` | `kuberay-operator` 1.7.0 — the `ray.io` CRDs and the controller. Namespaced (`singleNamespaceInstall: true`), so both the watch and the RBAC stay in `cms`. | +| `sonic-ray/chart/` | the `sonic-ray` chart: a `RayService` with a Triton in every worker pod and the forwarder as its Serve application, the ConfigMap carrying the forwarder, two metrics Services | +| `sonic-ray/chart/files/sonic_ray/serve_app.py` | the forwarder — one replica per pod, every unary RPC of `GRPCInferenceService` handed to the pod's Triton unchanged | +| `sonic-ray/helmrelease.yaml`, `sonic-ray/values.yaml` | the AF release: `dependsOn` the operator, values with the `triton:` block of the `supersonic` release's values | +| [`tests/sonic_ray/`](../../tests/sonic_ray), [`tests/manifests/test_ray.py`](../../tests/manifests/test_ray.py) | source-level checks of the forwarder; rendered-chart checks incl. parity with `apps/sonic/supersonic/values.yaml` | The chart lives here (like `apps/sonic/model-manager`) rather than being a raw `RayService` because of ordering: until the operator's chart has installed the @@ -57,6 +57,26 @@ else. Nothing else claims the resource, which is what leaves a pod without a replica idle and therefore reclaimable, and a replica without a pod pending — the request that grows the group. +### Ports + +Both containers share the pod's network namespace, so they cannot both use +Triton's defaults: + +| port | who | why it is where it is | +| ---- | -------------------- | -------------------------------------------------------------------------------------- | +| 8000 | Ray Serve HTTP proxy | KubeRay probes the proxy here; it does not move | +| 9000 | Ray Serve gRPC proxy | the inference entry point, behind the release's Service on 8001 | +| 8080 | Ray metrics | scraped by `sonic-ray-metrics` | +| 8100 | Triton HTTP | **moved** off Triton's default 8000, which Ray holds; serves only the kubelet's probes | +| 8001 | Triton gRPC | Triton's default; what the forwarder dials on localhost | +| 8002 | Triton metrics | scraped by `sonic-ray-triton-metrics` | + +Leaving Triton on 8000 is what makes it exit with `failed to start HTTP +service: Unavailable - Socket '0.0.0.0:8000' already in use`. The ports are +chart values (`triton.httpPort`, `triton.grpcPort`), and the chart refuses to +render if they collide with Ray's, if they disagree with the `--http-port` / +`--grpc-port` in `triton.args`, or if the args leave Triton on 8000. + ## What it serves and speaks Whatever Triton is pointed at. On the AF that is the models CMSSW ships: @@ -108,7 +128,7 @@ Two loops, both Ray's, nothing else in between: `terminationGracePeriodSeconds` against Triton's `--exit-timeout-secs` to drain (the chart refuses to render if the first is not larger). -One pair of numbers sizes both, because a replica *is* a pod. Raising the GPU +One pair of numbers sizes both, because a replica _is_ a pod. Raising the GPU ceiling is one edit in `sonic-ray/values.yaml`: ```yaml @@ -122,16 +142,16 @@ reclaims a pod that stays broken. ## How it lines up with SuperSONIC -| SuperSONIC (`supersonic`) | Ray (`sonic-ray`) | -| --- | --- | -| Triton on a per-site PVC or CVMFS, explicit load list | Triton on CVMFS, explicit load list — a plain `--model-repository` path | -| Envoy: gRPC entry point behind a `LoadBalancer` on `geddes-private-pool`, `ROUND_ROBIN` | Serve's gRPC proxy behind KubeRay's serve Service, same pool, port 8001 | -| `ingress.enabled: false` — private pool only | no ingress; the head is `ClusterIP`, dashboard by port-forward only | -| KEDA `ScaledObject` on a Prometheus expression, 1–10 pods | Ray Serve request-based autoscaling, 1–4 pods — see above | -| `nodeSelector: cms-af-prod=true` + the `hub.jupyter.org/dedicated` toleration | same, head and workers | -| model repository from a PVC or CVMFS | the cluster's `cvmfs` claim, mounted **read-only** | -| Triton Service labelled `scrape_metrics: "true"` | `sonic-ray-triton-metrics` (`nv_*`) and `sonic-ray-metrics` (Ray, incl. `ray_serve_*`), same label, `release="sonic-ray"` | -| Envoy's Lua rate limiter on `RepositoryIndex` | none; Serve's `maxOngoingRequests` back-pressure instead | +| SuperSONIC (`supersonic`) | Ray (`sonic-ray`) | +| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| Triton on a per-site PVC or CVMFS, explicit load list | Triton on CVMFS, explicit load list — a plain `--model-repository` path | +| Envoy: gRPC entry point behind a `LoadBalancer` on `geddes-private-pool`, `ROUND_ROBIN` | Serve's gRPC proxy behind KubeRay's serve Service, same pool, port 8001 | +| `ingress.enabled: false` — private pool only | no ingress; the head is `ClusterIP`, dashboard by port-forward only | +| KEDA `ScaledObject` on a Prometheus expression, 1–10 pods | Ray Serve request-based autoscaling, 1–4 pods — see above | +| `nodeSelector: cms-af-prod=true` + the `hub.jupyter.org/dedicated` toleration | same, head and workers | +| model repository from a PVC or CVMFS | the cluster's `cvmfs` claim, mounted **read-only** | +| Triton Service labelled `scrape_metrics: "true"` | `sonic-ray-triton-metrics` (`nv_*`) and `sonic-ray-metrics` (Ray, incl. `ray_serve_*`), same label, `release="sonic-ray"` | +| Envoy's Lua rate limiter on `RepositoryIndex` | none; Serve's `maxOngoingRequests` back-pressure instead | ## Using it diff --git a/apps/ray/sonic-ray/chart/templates/_helpers.tpl b/apps/ray/sonic-ray/chart/templates/_helpers.tpl index fa711959..af332f06 100644 --- a/apps/ray/sonic-ray/chart/templates/_helpers.tpl +++ b/apps/ray/sonic-ray/chart/templates/_helpers.tpl @@ -51,6 +51,9 @@ mounts, and the init container that pip-installs Triton's stubs. {{- define "sonic-ray.rayEnv" -}} - name: PYTHONPATH value: {{ printf "%s:%s" (include "sonic-ray.codeDir" .) (include "sonic-ray.depsDir" .) | quote }} +# Where the forwarder dials the Triton in its own pod. +- name: TRITON_GRPC + value: {{ printf "localhost:%d" (int .Values.triton.grpcPort) | quote }} {{- end -}} {{- define "sonic-ray.rayMounts" -}} @@ -115,6 +118,36 @@ Refuse to render what cannot work. {{- fail "triton.resources.limits must request exactly one nvidia.com/gpu: a pod is one Triton on one GPU." -}} {{- end -}} {{- $args := join " " .Values.triton.args -}} +{{/* +Triton and Ray share the pod's network namespace. Ray binds 8000 (Serve HTTP +proxy — KubeRay probes it there, so it is not the one that moves), 9000 +(Serve gRPC proxy) and 8080 (metrics); Triton must sit elsewhere and its args +must say the same numbers as the values, or the probes and the forwarder +would address a port nothing listens on. +*/}} +{{- range $name, $port := dict "httpPort" .Values.triton.httpPort "grpcPort" .Values.triton.grpcPort -}} + {{- if has (int $port) (list 8000 8080 9000) -}} + {{- fail (printf "triton.%s is %d, which Ray binds in this pod (8000 Serve HTTP, 9000 Serve gRPC, 8080 metrics). Triton would fail to bind it." $name (int $port)) -}} + {{- end -}} +{{- end -}} +{{- if eq (int .Values.triton.httpPort) (int .Values.triton.grpcPort) -}} + {{- fail "triton.httpPort and triton.grpcPort are the same port." -}} +{{- end -}} +{{- $http := regexFind "--http-port=[0-9]+" $args -}} +{{- if not $http -}} + {{- fail (printf "triton.args must pass --http-port=%d: Triton defaults to 8000, which Ray Serve's HTTP proxy already binds in this pod." (int .Values.triton.httpPort)) -}} +{{- end -}} +{{- if ne (trimPrefix "--http-port=" $http | int) (int .Values.triton.httpPort) -}} + {{- fail (printf "triton.args say %s but triton.httpPort is %d: the probes would address a port Triton does not listen on." $http (int .Values.triton.httpPort)) -}} +{{- end -}} +{{- $grpc := regexFind "--grpc-port=[0-9]+" $args -}} +{{- if $grpc -}} + {{- if ne (trimPrefix "--grpc-port=" $grpc | int) (int .Values.triton.grpcPort) -}} + {{- fail (printf "triton.args say %s but triton.grpcPort is %d: the forwarder would dial a port Triton does not listen on." $grpc (int .Values.triton.grpcPort)) -}} + {{- end -}} +{{- else if ne (int .Values.triton.grpcPort) 8001 -}} + {{- fail (printf "triton.grpcPort is %d but triton.args do not pass --grpc-port, so Triton listens on its default 8001." (int .Values.triton.grpcPort)) -}} +{{- end -}} {{- if not (contains .Values.triton.modelRepository.mountPath $args) -}} {{- fail (printf "triton.args never mention triton.modelRepository.mountPath (%s): Triton would not see the repository that is mounted." .Values.triton.modelRepository.mountPath) -}} {{- end -}} diff --git a/apps/ray/sonic-ray/chart/templates/rayservice.yaml b/apps/ray/sonic-ray/chart/templates/rayservice.yaml index d791e91e..67c74495 100644 --- a/apps/ray/sonic-ray/chart/templates/rayservice.yaml +++ b/apps/ray/sonic-ray/chart/templates/rayservice.yaml @@ -184,8 +184,10 @@ spec: args: {{- toYaml .Values.triton.args | nindent 18 }} ports: - - { name: http, containerPort: 8000 } - - { name: grpc, containerPort: 8001 } + # Named, because the probes address http by name. Neither is + # Triton's default: Ray has 8000 in this pod (see values). + - { name: http, containerPort: {{ .Values.triton.httpPort }} } + - { name: grpc, containerPort: {{ .Values.triton.grpcPort }} } - { name: triton-metrics, containerPort: 8002 } resources: {{- toYaml .Values.triton.resources | nindent 18 }} diff --git a/apps/ray/sonic-ray/chart/values.yaml b/apps/ray/sonic-ray/chart/values.yaml index ac1b265b..ae4e52c9 100644 --- a/apps/ray/sonic-ray/chart/values.yaml +++ b/apps/ray/sonic-ray/chart/values.yaml @@ -99,12 +99,25 @@ triton: repository: nvcr.io/nvidia/tritonserver tag: "26.04-py3" pullPolicy: IfNotPresent - # -- Passed to the Triton container as-is; the model repository path in args - # must be modelRepository.mountPath. + # -- Passed to the Triton container as-is. The model repository path in args + # must be modelRepository.mountPath, and the ports in args must be the ones + # below — the chart refuses to render otherwise. command: ["tritonserver"] args: - --model-repository=/models + - --http-port=8100 - --exit-timeout-secs=60 + # -- Ports Triton listens on. The Ray container shares this pod's network + # namespace, so these must avoid the ports Ray binds there: 8000 (Serve's + # HTTP proxy), 9000 (Serve's gRPC proxy), 8080 (metrics). HTTP therefore + # cannot be Triton's default 8000 — it would fail to bind with "Socket + # '0.0.0.0:8000' already in use" — and Serve's proxy is the one that has to + # keep 8000, since KubeRay probes the proxy there. Nothing outside the pod + # reaches either of these: inference arrives at Serve's gRPC proxy, which + # forwards to grpcPort on localhost, and httpPort serves only the kubelet's + # probes. + httpPort: 8100 + grpcPort: 8001 # -- One GPU per Triton is the unit of scaling: a pod is one Triton, and a # Serve replica is one pod. resources: diff --git a/apps/ray/sonic-ray/values.yaml b/apps/ray/sonic-ray/values.yaml index 8318f369..2e7134e8 100644 --- a/apps/ray/sonic-ray/values.yaml +++ b/apps/ray/sonic-ray/values.yaml @@ -14,6 +14,7 @@ triton: args: - | /opt/tritonserver/bin/tritonserver \ + --http-port=8100 \ --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoBTag/Combined/data/models/ \ --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoEgamma/EgammaPhotonProducers/data/models/ \ --model-repository=/cvmfs/cms.cern.ch/el9_amd64_gcc13/cms/cmssw/CMSSW_17_0_0_pre2/external/el9_amd64_gcc13/data/RecoTauTag/TrainingFiles/data/DeepTauIdSONIC/ \ @@ -55,6 +56,11 @@ triton: failureThreshold: 10 successThreshold: 1 httpGet: { path: /v2/health/ready, port: http } + # Triton's HTTP moves off its default 8000: the Ray container in this pod + # binds that for Ray Serve's HTTP proxy, and containers of a pod share one + # network namespace. gRPC keeps 8001, which is what the forwarder dials. + httpPort: 8100 + grpcPort: 8001 # The cluster's CVMFS claim (apps/infrastructure/cvmfs-pvc.yaml), read-only. # HostToContainer: CVMFS repositories are autofs mounts on the node, and the # pod must see the ones mounted after it started. diff --git a/tests/manifests/test_ray.py b/tests/manifests/test_ray.py index e24b2b27..d9987537 100644 --- a/tests/manifests/test_ray.py +++ b/tests/manifests/test_ray.py @@ -271,12 +271,53 @@ def test_rendered_triton_is_the_one_in_values(worker_group, values): assert triton["readinessProbe"] == values["triton"]["readinessProbe"] assert triton["startupProbe"] == values["triton"]["startupProbe"] assert {p["name"]: p["containerPort"] for p in triton["ports"]} == { - "http": 8000, - "grpc": 8001, + "http": values["triton"]["httpPort"], + "grpc": values["triton"]["grpcPort"], "triton-metrics": 8002, } +def test_triton_and_ray_do_not_fight_over_a_port(worker_group, values): + """Containers of a pod share one network namespace. Triton's defaults + collide with Ray Serve's HTTP proxy on 8000 — it fails to start with + "Socket '0.0.0.0:8000' already in use" — so Triton moves, and Serve keeps + 8000 because KubeRay probes the proxy there. The args must name the same + ports as the values, or the probes and the forwarder address nothing.""" + declared = [ + (c["name"], p["name"], p["containerPort"]) + for c in worker_group["template"]["spec"]["containers"] + for p in c.get("ports", []) + ] + numbers = [port for _, _, port in declared] + assert len(numbers) == len(set(numbers)), ( + f"two containers claim one port: {declared}" + ) + + ray_ports = {port for owner, _, port in declared if owner == "ray-worker"} + assert 8000 in ray_ports, ( + "Serve's HTTP proxy must keep 8000: KubeRay probes it there" + ) + triton = values["triton"] + assert triton["httpPort"] != 8000 + args = triton["args"][0].split() + assert f"--http-port={triton['httpPort']}" in args + # gRPC is left at Triton's default, which is what the forwarder dials. + assert triton["grpcPort"] == 8001 + assert not [a for a in args if a.startswith("--grpc-port=")] + + +def test_the_forwarder_is_told_where_its_triton_is(head_pod, worker_group, values): + """The forwarder dials TRITON_GRPC; unset, it falls back to a default that + would be wrong the moment grpcPort changed.""" + expected = f"localhost:{values['triton']['grpcPort']}" + for template, name in ( + (head_pod, "ray-head"), + (worker_group["template"], "ray-worker"), + ): + assert env_of(container(template, name))["TRITON_GRPC"] == expected + assert f'"TRITON_GRPC", "{expected}"' in SERVE_APP.read_text() + + def test_models_come_from_cvmfs_read_only(worker_group, values): """The cluster's CVMFS claim, mounted read-only with host-to-container propagation (CVMFS repositories are autofs mounts on the node), and every @@ -501,6 +542,9 @@ def test_metrics_services_select_labels_kuberay_leaves_alone( [ ("triton.modelRepository.claimName=", "claimName is required"), ("replicas.min=5", "replicas.min exceeds replicas.max"), + ("triton.httpPort=8000", "which Ray binds in this pod"), + ("triton.httpPort=8200", "but triton.httpPort is 8200"), + ("triton.grpcPort=8500", "do not pass --grpc-port"), ("triton.resources.limits.nvidia\\.com/gpu=2", "exactly one nvidia.com/gpu"), ( "ray.worker.terminationGracePeriodSeconds=30",