Triton on Ray: Ray Serve's gRPC proxy in front of Triton

.github/workflows/validate-manifests.sh

@@ -12,6 +12,10 @@
 #      the chart version, HelmRepository URL and valuesFrom-ConfigMap values
 #      resolved from that same stream — this is what catches breaking chart
 #      schema changes when a chart version is bumped (e.g. by Renovate).
+#      Charts that live in this repository (sourced from its GitRepository,
+#      path ./apps/...) are rendered from the working tree with the same
+#      values, so a broken template or a values/template mismatch fails here
+#      rather than in the cluster.
 #   5. `promtool check rules` on Prometheus alerting/recording rules embedded
 #      in Helm values — a typo'd PromQL expression otherwise deploys silently
 #      and the alert simply never fires.
@@ -114,25 +118,33 @@ validate_helmreleases() {
 		src_name=$(yq "select(.kind==\"HelmRelease\" and .metadata.name==\"$name\") | .spec.chart.spec.sourceRef.name" "$rendered")
 		src_kind=$(yq "select(.kind==\"HelmRelease\" and .metadata.name==\"$name\") | .spec.chart.spec.sourceRef.kind" "$rendered")
 
-		if [[ "$src_kind" != "HelmRepository" ]]; then
+		# A chart in this repository is rendered from the working tree. Only
+		# ./apps/... paths qualify: other GitRepository sources (supersonic-dev,
+		# servicex-dev) point at charts in someone else's repository.
+		local_chart=""
+		if [[ "$src_kind" == "GitRepository" && "$chart" == ./apps/* && -f "$chart/Chart.yaml" ]]; then
+			local_chart="$chart"
+		elif [[ "$src_kind" != "HelmRepository" ]]; then
 			echo "  skip ${name}: chart sourced from ${src_kind} '${src_name}' (no registry version to validate)"
 			continue
 		fi
 
-		if [[ -z "$version" || "$version" == "null" ]]; then
-			echo "✗ ${name}: chart version is not pinned (omitted version = Flux silently tracks latest)" >&2
-			failed=1
-			continue
-		fi
+		if [[ -z "$local_chart" ]]; then
+			if [[ -z "$version" || "$version" == "null" ]]; then
+				echo "✗ ${name}: chart version is not pinned (omitted version = Flux silently tracks latest)" >&2
+				failed=1
+				continue
+			fi
 
-		repo_line=$(grep -m1 "^${src_name}|" "$repos_file" || true)
-		if [[ -z "$repo_line" ]]; then
-			echo "✗ ${name}: HelmRepository '${src_name}' not found in any rendered environment" >&2
-			failed=1
-			continue
+			repo_line=$(grep -m1 "^${src_name}|" "$repos_file" || true)
+			if [[ -z "$repo_line" ]]; then
+				echo "✗ ${name}: HelmRepository '${src_name}' not found in any rendered environment" >&2
+				failed=1
+				continue
+			fi
+			repo_type=$(cut -d'|' -f2 <<<"$repo_line")
+			repo_url=$(cut -d'|' -f3- <<<"$repo_line")
 		fi
-		repo_type=$(cut -d'|' -f2 <<<"$repo_line")
-		repo_url=$(cut -d'|' -f3- <<<"$repo_line")
 
 		# Resolve valuesFrom ConfigMaps (generated by kustomize from values.yaml
 		# files) into temp files, in order.
@@ -167,6 +179,19 @@ validate_helmreleases() {
 			done
 		fi
 
+		if [[ -n "$local_chart" ]]; then
+			echo "  helm template ${name} (${local_chart}, from this repository)"
+			if ! out=$(helm template "$name" "$local_chart" \
+				--kube-version "$KUBE_VERSION" \
+				--namespace cms \
+				${values_args[@]+"${values_args[@]}"} 2>&1); then
+				printf '%s\n' "$out" >&2
+				echo "✗ ${name}: helm template failed" >&2
+				failed=1
+			fi
+			continue
+		fi
+
 		local fingerprint="${repo_url}|${chart}|${version}|${vhash}"
 		if [[ "$seen_releases" == *"$fingerprint"* ]]; then
 			continue

README.md

@@ -71,6 +71,8 @@ Whether each component on the cluster is running what is on `main`
 ![supersonic][experimental-sonic-supersonic]
 ![supersonic-dev][experimental-sonic-supersonic-dev]
 ![model-manager][experimental-sonic-model-manager]
+![kuberay-operator][experimental-ray-operator]
+![sonic-ray][experimental-ray-sonic-ray]
 
 **Images** — `purdue-af` is released on its own semver stream and pinned at
 ![AF image][af-image-version]. `agentic-interface` is auto-versioned (every
@@ -136,6 +138,8 @@ How a change reaches the cluster, version rules and rollback:
 [experimental-sonic-model-manager]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/experimental-sonic-model-manager.json
 [experimental-sonic-supersonic]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/experimental-sonic-supersonic.json
 [experimental-sonic-supersonic-dev]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/experimental-sonic-supersonic-dev.json
+[experimental-ray-operator]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/experimental-ray-operator.json
+[experimental-ray-sonic-ray]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/experimental-ray-sonic-ray.json
 [image-purdue-af]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/image-purdue-af.json
 [image-agentic-interface]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/image-agentic-interface.json
 [image-af-pod-monitor]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/PurdueAF/purdue-af/status/badges/image-af-pod-monitor.json

apps/ray/README.md

@@ -0,0 +1,188 @@
+# Ray on the Analysis Facility
+
+**Triton on Ray**: every worker pod carries NVIDIA's Triton Inference Server,
+configured exactly as in the `supersonic` release (`apps/sonic/supersonic`) —
+same image, arguments, resources, model repository — and **Ray Serve's gRPC proxy carries Triton's protocol** to it.
+Serve speaks that protocol because it is handed Triton's own generated
+servicer; the only code of ours is a forwarder that passes each RPC from the
+proxy to the Triton in its pod. Serve counts every request on the way through,
+sizes the deployment from that, and the Ray autoscaler adds a GPU pod for each
+replica with nowhere to go.
+
+No custom image, no protocol code, no model code: official Ray, official
+Triton, ~100 lines of glue shipped as a ConfigMap, Triton's Python stubs
+pip-installed by an init container.
+
+| path | what it is |
+| --- | --- |
+| `helmrepo.yaml` | `HelmRepository` for the KubeRay charts |
+| `operator/` | `kuberay-operator` 1.7.0 — the `ray.io` CRDs and the controller. Namespaced (`singleNamespaceInstall: true`), so both the watch and the RBAC stay in `cms`. |
+| `sonic-ray/chart/` | the `sonic-ray` chart: a `RayService` with a Triton in every worker pod and the forwarder as its Serve application, the ConfigMap carrying the forwarder, two metrics Services |
+| `sonic-ray/chart/files/sonic_ray/serve_app.py` | the forwarder — one replica per pod, every unary RPC of `GRPCInferenceService` handed to the pod's Triton unchanged |
+| `sonic-ray/helmrelease.yaml`, `sonic-ray/values.yaml` | the AF release: `dependsOn` the operator, values with the `triton:` block of the `supersonic` release's values |
+| [`tests/sonic_ray/`](../../tests/sonic_ray), [`tests/manifests/test_ray.py`](../../tests/manifests/test_ray.py) | source-level checks of the forwarder; rendered-chart checks incl. parity with `apps/sonic/supersonic/values.yaml` |
+
+The chart lives here (like `apps/sonic/model-manager`) rather than being a raw
+`RayService` because of ordering: until the operator's chart has installed the
+`ray.io` CRDs that is an unknown kind, and kustomize-controller aborts an apply
+on the first one it meets — on a fresh cluster, before the HelmRelease that
+would install them. `dependsOn: kuberay-operator` is the fix, and a
+`HelmRelease` is the only object that can carry it.
+
+## Shape
+
+```
+     clients ──▶ sonic-ray-serve (LoadBalancer, private pool) :8001
+                    │  Triton gRPC: ModelInfer, ModelMetadata, …
+                    ▼
+     Serve gRPC proxy (Triton's servicer) on head and every worker
+                    │  counted, balanced, autoscaled by Serve
+                    ▼
+   ┌─────────────────────────────────────────────────┐   × 1…4 pods
+   │ worker pod                                      │
+   │   ray-worker  raylet advertising triton: 1,     │
+   │               proxy, TritonProxy replica ──┐    │
+   │   triton      1 GPU, 4 CPU, 16G, /cvmfs ro ◀┘   │  localhost:8001
+   └─────────────────────────────────────────────────┘
+                    ▲ replica demand
+   ┌─────────────────────────────────────────────────┐
+   │ head pod: Serve controller, Ray autoscaler      │  (0 CPUs for work, no GPU)
+   └─────────────────────────────────────────────────┘
+```
+
+A **pod is one Triton on one GPU**, the unit SuperSONIC scales by too. A
+**replica is one pod**: every worker advertises one `triton` resource and
+every replica claims one, so a replica lands next to its Triton and nowhere
+else. Nothing else claims the resource, which is what leaves a pod without a
+replica idle and therefore reclaimable, and a replica without a pod pending —
+the request that grows the group.
+
+## What it serves and speaks
+
+Whatever Triton is pointed at. On the AF that is the models CMSSW ships:
+`sonic-ray/values.yaml` mounts the cluster's CVMFS claim read-only at `/cvmfs`
+and gives Triton four `--model-repository` directories inside a CMSSW release
+(`CMSSW_17_0_0_pre2` today — RecoBTag, RecoEgamma, RecoTauTag, RecoMET) with
+an explicit load list:
+
+`deepmet`, `deeptau_2018v2p5`, `particleNetFromMiniAODAK4CHSCentral`,
+`particleNetFromMiniAODAK4PuppiCentral`, `particleNetFromMiniAODAK4PuppiForward`,
+`particleNetFromMiniAODAK8`, `particlenet_AK8_MD-2prong_PT`,
+`particlenet_AK8_MassRegression_PT`, `particlenet_PT`,
+`unifiedparticletransformer_AK4_V01`.
+
+Nothing is uploaded anywhere and there is no model manager: a new CMSSW
+release, or a different model set, is a path change in the values. Every
+backend, `config.pbtxt` semantics, dynamic batching and the repository index
+work as in any Triton, because it is Triton. The first load from CVMFS pulls
+the files over the network into the node's cache, so the startup probe allows
+four minutes.
+
+The wire protocol is Triton's gRPC. HTTP is **not** carried (Serve's HTTP
+proxy on 8000 answers only its own `/-/healthz` and `/-/routes`); Triton's
+HTTP port stays inside the pod. CMSSW's `TritonClient` speaks gRPC, so
+`cmsRun` jobs point at `sonic-ray-serve:8001` as at any Triton endpoint — the
+port is Triton's conventional one on purpose. `tritonclient.grpc` works the
+same way.
+
+The one RPC not forwarded is `ModelStreamInfer`, Triton's bidirectional
+stream: Serve's proxy carries unary and server-streaming calls only. CMSSW
+uses the unary `ModelInfer`.
+
+## Autoscaling
+
+Two loops, both Ray's, nothing else in between:
+
+1. **Ray Serve** sizes the deployment from the requests its gRPC proxy
+   forwards. When the average number in flight per replica exceeds
+   `serve.targetOngoingRequests` (16) for `upscaleDelayS` (10 s) it adds a
+   replica; when it falls well below for `downscaleDelayS` (300 s) it removes
+   one, giving in-flight requests `gracefulShutdownTimeoutS` (60 s). Bounds are
+   `replicas.min`/`max` (1/4 on the AF).
+2. **The Ray autoscaler** sizes the cluster. A new replica needs a `triton`
+   resource; if no worker has one free, that is a pending request and the
+   autoscaler adds a pod to `gpu-group` (ceiling: the same `replicas.max`; the
+   group's own floor is 0, since a pod with a replica on it is never idle and
+   Serve's minimum therefore keeps pods alive). A worker whose replica is gone idles for
+   `idleTimeoutSeconds` (60 s) and is reclaimed; the pod then gets
+   `terminationGracePeriodSeconds` against Triton's `--exit-timeout-secs` to
+   drain (the chart refuses to render if the first is not larger).
+
+One pair of numbers sizes both, because a replica *is* a pod. Raising the GPU
+ceiling is one edit in `sonic-ray/values.yaml`:
+
+```yaml
+replicas: { min: 1, max: 8 }
+```
+
+A replica only becomes ready once its Triton answers `ServerReady`, and it
+polls `ServerLive` as its health check, so Serve never routes to a pod whose
+Triton is still loading or has died — Serve restarts the replica, and Ray
+reclaims a pod that stays broken.
+
+## How it lines up with SuperSONIC
+
+| SuperSONIC (`supersonic`) | Ray (`sonic-ray`) |
+| --- | --- |
+| Triton on a per-site PVC or CVMFS, explicit load list | Triton on CVMFS, explicit load list — a plain `--model-repository` path |
+| Envoy: gRPC entry point behind a `LoadBalancer` on `geddes-private-pool`, `ROUND_ROBIN` | Serve's gRPC proxy behind KubeRay's serve Service, same pool, port 8001 |
+| `ingress.enabled: false` — private pool only | no ingress; the head is `ClusterIP`, dashboard by port-forward only |
+| KEDA `ScaledObject` on a Prometheus expression, 1–10 pods | Ray Serve request-based autoscaling, 1–4 pods — see above |
+| `nodeSelector: cms-af-prod=true` + the `hub.jupyter.org/dedicated` toleration | same, head and workers |
+| model repository from a PVC or CVMFS | the cluster's `cvmfs` claim, mounted **read-only** |
+| Triton Service labelled `scrape_metrics: "true"` | `sonic-ray-triton-metrics` (`nv_*`) and `sonic-ray-metrics` (Ray, incl. `ray_serve_*`), same label, `release="sonic-ray"` |
+| Envoy's Lua rate limiter on `RepositoryIndex` | none; Serve's `maxOngoingRequests` back-pressure instead |
+
+## Using it
+
+```bash
+kubectl -n cms get svc sonic-ray-serve            # MetalLB address on the private pool
+SONIC=<address>:8001
+```
+
+CMSSW clients point at `$SONIC`, exactly as they point at the supersonic
+release's Envoy address. From Python:
+
+```python
+import tritonclient.grpc as grpcclient
+
+client = grpcclient.InferenceServerClient("<address>:8001")
+client.is_server_ready()
+client.get_model_repository_index()
+```
+
+The Ray dashboard, for Serve and autoscaler state:
+
+```bash
+kubectl -n cms port-forward svc/sonic-ray-head-svc 8265:8265
+```
+
+## What is not an image
+
+The Ray containers run `rayproject/ray:2.52.0-py312-cpu` (through the geddes
+Docker Hub proxy cache) exactly as published; the Triton container runs the
+image the values name (the chart default is `nvcr.io/nvidia/tritonserver`;
+the AF values use the lighter `docexoty/tritonserver:light`). Two things
+are added at deploy time instead of build time:
+
+- **the forwarder** — `files/sonic_ray/*.py` become the `sonic-ray-code`
+  ConfigMap, mounted at `/serve_app/sonic_ray` on head and workers. Its hash
+  is annotated onto both pod templates, so a code change rolls the cluster.
+- **Triton's Python stubs** — `python.pip` (`tritonclient==2.48.0`, the last
+  release whose generated stubs match the protobuf 4 in the Ray image, plus
+  `python-rapidjson`) is pip-installed `--no-deps --target` into an emptyDir
+  by an init container on every pod, and that directory is on `PYTHONPATH`.
+  Serve's proxies import the servicer from it at startup, on every node,
+  which is why a `runtime_env` (replicas only) would not do.
+
+The price is a small pip download per pod start and a dependency on PyPI
+being reachable from the nodes — chosen over maintaining an image.
+
+## Cost
+
+One GPU idles (`replicas.min: 1`) on the same `cms-af-prod` nodes
+SuperSONIC and the user sessions compete for. An upgrade costs a second set
+for its duration: `upgradeStrategy: NewCluster` brings a second cluster up
+before cutting over, and if no GPU is free it waits while the old one keeps
+serving. A GPU node here has 128 cores, so the two extra CPUs the Ray
+container adds to each pod change nothing about what fits.

apps/ray/helmrepo.yaml

@@ -0,0 +1,7 @@
+apiVersion: source.toolkit.fluxcd.io/v1beta2
+kind: HelmRepository
+metadata:
+  name: kuberay
+spec:
+  interval: 1h
+  url: https://ray-project.github.io/kuberay-helm/

apps/ray/operator/helmrelease.yaml

@@ -0,0 +1,29 @@
+apiVersion: helm.toolkit.fluxcd.io/v2beta1
+kind: HelmRelease
+metadata:
+  name: kuberay-operator
+spec:
+  suspend: false
+  interval: 1m
+  chart:
+    spec:
+      chart: kuberay-operator
+      version: "1.7.0"
+      sourceRef:
+        kind: HelmRepository
+        name: kuberay
+      interval: 1m
+  install:
+    # The ray.io CRDs (RayCluster/RayJob/RayService/RayCronJob) ship in the
+    # chart's crds/ directory; nothing else in the cluster installs them.
+    crds: Create
+    remediation:
+      retries: -1
+  upgrade:
+    crds: CreateReplace
+    remediation:
+      retries: -1
+  valuesFrom:
+    - kind: ConfigMap
+      name: kuberay-operator-config
+      valuesKey: values.yaml

apps/ray/operator/values.yaml

@@ -0,0 +1,24 @@
+# KubeRay operator for the AF. Deployed only so RayCluster CRs can be
+# reconciled — see apps/ray/sonic-ray for the cluster itself.
+
+# Watch the release namespace (cms) only, and create namespaced Role/RoleBinding
+# instead of cluster-wide ClusterRole/ClusterRoleBinding. The AF has no Ray
+# workloads outside cms, so there is no reason to grant cluster scope.
+singleNamespaceInstall: true
+
+# The operator is a controller, not a data-plane component: keep it off the
+# GPU nodes' dedicated taint and let it land anywhere in the cluster.
+resources:
+  limits:
+    cpu: 500m
+    memory: 512Mi
+  requests:
+    cpu: 100m
+    memory: 256Mi
+
+metrics:
+  enabled: true
+  # The AF Prometheus scrapes by Service label (scrape_metrics: "true"),
+  # not through prometheus-operator ServiceMonitors.
+  serviceMonitor:
+    enabled: false

apps/ray/sonic-ray/chart/Chart.yaml

@@ -0,0 +1,9 @@
+apiVersion: v2
+name: sonic-ray
+description: Triton Inference Server on Ray — Ray Serve's gRPC proxy carries Triton's protocol to a Triton in every GPU pod, and Ray autoscales the pods from the requests.
+type: application
+version: 0.1.0
+appVersion: "26.04"
+home: https://github.com/PurdueAF/purdue-af/tree/main/apps/ray
+annotations:
+  artifacthub.io/category: ai-machine-learning

apps/ray/sonic-ray/chart/files/sonic_ray/__init__.py

@@ -0,0 +1,12 @@
+"""sonic-ray: Triton on Ray, with Ray Serve carrying the gRPC traffic.
+
+One module, ``serve_app``: a Ray Serve deployment that forwards every unary
+RPC of Triton's ``GRPCInferenceService`` to the Triton running beside it in
+the same pod. Ray Serve's gRPC proxy speaks Triton's protocol because it is
+handed Triton's own generated servicer (from the ``tritonclient`` package);
+Triton does every bit of the inference. Nothing here parses a request.
+
+The file ships to the cluster as a ConfigMap rendered by the chart (see
+templates/configmap.yaml) and lands on PYTHONPATH in the stock Ray image;
+there is no custom image.
+"""