Grafana dashboard for Triton on Ray

[kondratyevd]

What

A Grafana dashboard for the sonic-ray release, provisioned into the AF Grafana next to the existing ones: Triton on Ray (uid sonic-ray).

Follow-up to #212 and #218 (both merged): one dashboard, one line of Flux wiring, and its tests.

Rebuilt on main after the public/private Grafana split (#190). It goes to the public instance, where the SuperSONIC dashboard already lives — that is the one with anonymous viewer access, which is what you want while watching a benchmark. Moving it to the private instance is a one-line change if you would rather.

The existing SuperSONIC dashboard cannot show this release: several of its panels filter app=~"supersonic", and this release's Services are labelled app=sonic-ray. Rather than loosen a production dashboard, this is a separate one, scoped by the same release / namespace variables.

Panels

Ray Serve — the request path

panel	what it answers
Replicas serving / Requests per second / In flight per replica	the three numbers that describe the current state; the third is the signal Serve autoscales on, coloured against the target
Requests per second by RPC	Triton's own RPC names as the proxy sees them (method is a declared tag key), so metadata chatter is visible separately from ModelInfer
Serve latency	p50/p95/p99/mean at the replica, from the processing-latency histogram
Autoscaling	replicas, in-flight and queued together — queued rising while replicas are flat is the group falling behind
Errors per second	proxy-level and deployment-level; a replica whose Triton died shows here first
Ray overhead	Serve's mean latency minus Triton's own mean request duration. The number this whole deployment exists to measure

Triton — the inference: inferences/s by model, the queue/input/infer/output latency breakdown (the SuperSONIC dashboard's queries, scoped to this release), pending requests per server, GPU utilization and memory.

How the queries were built

Every series name was taken from a source, not from memory:

• Ray Serve metrics from Ray 2.52's own definitions (ray_serve_deployment_replica_healthy, ..._processing_latency_ms, ..._replica_processing_queries, ..._queued_queries, ..._error_counter_total, and ray_serve_num_grpc_requests_total / ..._error_requests_total, which the proxy builds as serve_num_{protocol}_requests).
• method is a declared tag key of that proxy counter; pod, release, namespace, app and component are attached by the AF Prometheus itself (the af-pod-monitor relabel rules). Panels group only by those, never by Ray's internal deployment/replica tags, which this Prometheus does not key on.
• The nv_* series and the datasource uid prometheus match what the SuperSONIC dashboard already plots here.

Since the split, the tests also pin which instance provisions it: present in the public ConfigMap, absent from the private one, and each instance mounting the map it is named for. tests/manifests/test_sonic_ray_dashboard.py holds that, plus: an allowlist of series with a note on where each was verified, every target scoped to both variables, grouping only by labels that exist, $__rate_interval windows, every division guarded so an idle release cannot render ±∞ (the failure the pixi panels already learned), the overhead panel converting Triton's microseconds to Serve's milliseconds and not clipping at zero, and no two panels overlapping on the grid. Two of those tests failed on the first draft and caught real bugs.

Verified

• pytest tests 981 passing, validate-manifests.sh green, ruff and prettier clean.
• Not rendered against a live Grafana or a live Prometheus. The panels stay empty until the release actually runs, which needs Fix Triton failing to start: move its HTTP port off Ray Serve's 8000 #218 (Triton currently fails to start). That first deploy is also what confirms the Ray metrics are scraped at all: KubeRay is expected to export them on the port named metrics (8080), which sonic-ray-metrics selects, but no pod has run yet to prove it.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!