sunilmehta.si ↗   ·   LinkedIn ↗   ·   Building at ORGN   ·   Connect on X

I'm Sunil Mehta, a lead DevOps, MLOps and platform engineer and founding lead at ORGN. I work across cloud infrastructure, confidential computing, identity, and the systems that turn AI capabilities into dependable products.

My scope runs from terraform plan to production traffic: Kubernetes and GitOps, confidential compute, and secure delivery pipelines.

These highlights describe my professional experience. Most of the underlying work lives in private organization repositories.

Kubernetes Secure MLOps Pipeline — An MLOps pipeline on Kubernetes where every model that reaches production is reproducible, evaluated against the model it replaces, signed and verified at deploy time. Kubeflow Pipelines and MLflow train and gate; models ship as signed KServe images with SLSA provenance and an ML-BOM, and Kyverno refuses anything else at admission. One Helm chart is delivered to dev, staging and prod by Argo CD; live drift and accuracy alerts start retraining; a load drill holds the serving objectives. A real kind cluster runs 24 admission, network, monitoring, feedback, load and GitOps checks in CI on every push.

Explore the governance and threat model, serving and its drills, and environments and GitOps.

Kubernetes GPU Scheduler Benchmark — Which GPU scheduler should a platform run? Replays the same AI workload (training gangs, fine-tunes, inference, notebooks) through the default kube-scheduler, Kueue, Volcano and KAI Scheduler, first in a reference simulation and then on real schedulers in CI (kind + kwok GPU nodes), and measures utilisation, fragmentation, queue wait, fairness and cost, up to a 1,024-GPU fleet. Finding: out of the box only KAI bin-packs GPU pods by their GPUs; one setting fixes each of the others, and on the real Volcano scheduler it made the same trace 26% faster and 26% cheaper per useful GPU-hour. Every cluster version is pinned in one file, and a report says what upstream has released since and when it is old enough to adopt.

Explore the results, real-scheduler checks, and methodology.

Kubernetes Inference Routing Benchmark — How much does the router in front of LLM servers matter? Replays the same chat and RAG traffic through a Kubernetes Service and the llm-d Router (the Gateway API Inference Extension's endpoint picker) with load, prefix-cache and combined scoring, on kind clusters with simulated vLLM pods. Every router runs eight times, and only differences that held in every run are claimed. Finding: prefix-aware routing lifts the pods' cache hits from 64% to 80% and cuts median time to first token by 27%, but sends most requests to two of four pods, so load-aware routing keeps the best tail. Tuning the load scorer's weight or threshold did not spread the work; swapping in the router's active-request scorer did, beating prefix-plus-load on median, p95 and p99 time to first token in all eight runs, and with the prefix at weight 2 it keeps 83% cache hits with the best p95 of any router measured; on eight pods the prefix routers' pile-up doubles while it stays even, and on a control trace with nothing to cache it keeps its whole latency advantage, which shows the gain is the load scoring. Reading the pods' real cache contents from their KV-cache events, instead of the picker's estimate, finds two to three more points of cache hits in every run and a better tail, for a tokenizer round trip at the median; six more datasets place that: the gain needs a load scorer strong enough to move prompts off the pod that holds them, on eight pods the real picture spreads the best approximate router's pile-up, the median cost is the extra hop rather than tokenizer CPU and vanishes behind long prompts, and speculative indexing helps on four pods and hurts on eight. Under load the router matters more and the routers differ less: at three times the trace's rate a plain Service's median is 3.6x the best router's and a tenth of its requests miss the objective, at four times it passes a second while every router stays under 170 ms, and past that point only having a router at all holds. The scorers' own formulas, read from the router's source, explain each result, and a test checks every claim against the runs.

Explore the results, methodology, and the releases.

Kubernetes Inference Autoscaling Benchmark — What should an autoscaler follow when it scales LLM inference, and can it hold a latency objective at all? Replays the same bursty chat trace through a model of a vLLM deployment scaled by the Kubernetes HorizontalPodAutoscaler's own algorithm on four signals (slot utilization, requests in the system, queue depth, request rate), eight runs per policy, with only differences that held in every run claimed. Finding: no default autoscaler holds a 2 second first-token objective on a ramp from 3 to 20 requests per second in 30 seconds; the best signal serves 80.3% of requests in time at 74% of the cost of a static pool sized for the peak. The miss is pod startup first and the delay in deciding second, not the target or the scale-up limit: pods that start in one second with a decision every second serve every request at 62% of the static pool's cost, and keeping at least 12 pods lifts the default to 93.5% for about 3% more. A simulation with the controller modelled step by step, seven datasets, and a test that checks every number in the README against the saved results. The model is then checked against the real HPA, Prometheus and prometheus-adapter on a kind cluster (twelve 30-minute runs): medians and scaling decisions agree, the tail does not, and giving each pod its own queue, as behind a Service, moves the model toward the cluster and reproduces its ranking of the signals. The seven datasets are re-run that way too, and the findings that depend on the shared queue are named.

Explore the results, methodology, the cluster comparison and the release.

Agentic Code Review Tool — Scanner-grounded AI review for the whole pull request: 128 rules across app code, Kubernetes, Docker and Compose, Terraform, GitHub Actions, lock files and AI/agent code (OWASP LLM Top 10, MCP configs, agent skills and plugins, coding-agent settings), swept over Argo CD, Kubeflow Pipelines, KServe, MLflow and flytekit before each release so the false positives are fixed first (4,654 findings to 2,315 in one cycle, every one removed a test input, a fixture or the name of a secret, then 1,984 to 1,942 over KubeRay, Kueue, Kubeflow Trainer, Volcano, KAI Scheduler, llm-d and production-stack, removing CRD schemas read as pod specs and code its authors had already marked //nolint:gosec), with Go now covered by the weak-hash, deprecated-TLS, JWT, CORS and cookie rules too, including the credential-stealing workflow shape of this year's GhostAction attacks, checkout tokens shipped inside workflow artifacts, RBAC escalation paths short of *, model archives extracted without a path filter, HTTP calls that wait forever, Go reviewed as fully as Python (SQL built with fmt.Sprintf, sh -c handed a built command, servers with no header timeout), Dependabot configurations that propose a release the day it is published, and lock files that still trust the compromised emSign root, only on changed lines, with an optional Claude or local-LLM agent that verifies findings using read-only, repository-confined tools. Reports to GitHub, GitLab, SARIF and JUnit.

See a live AI review on a pull request, the rule catalogue, and the threat model.

AI DevSecOps CI/CD — A reference pipeline for shipping LLM and AI-agent services safely: prompt-injection tests mapped to the OWASP LLM Top 10, SBOM and AI-BOM generation, Trivy and secret scanning, keyless cosign signing with attestations, and Kyverno admission policies with an offline validator mirroring them in CI.

Explore the threat model, AI security test corpus, and workflows.

Jev Inference Router — An observable decision layer for local GPU inference: TypeSafe Jev chooses the handler, Python applies the policy, and local Qwen generates the answer, with visible probabilities, retrieval, Prometheus metrics, and reproducible evaluations against rule-based and classifier baselines.

Explore the architecture, evaluations, and operations guide.

LLM Inference Platform — GPU-backed LLM serving on Apple Silicon with vLLM-Metal, a Python streaming gateway, Prometheus/Grafana monitoring, reproducible benchmarks, and Kubernetes deployment templates.

Explore the architecture, operational tests, and benchmark evidence.

Infrastructure   Kubernetes · Terraform · Argo CD · Helm · GCP · AWS · DigitalOcean · Cloudflare
MLOps & serving   Kubeflow Pipelines · MLflow · KServe · vLLM
Systems & services   Go · Rust · TypeScript · Python · PostgreSQL · Redis
GPU scheduling   Kueue · Volcano · KAI Scheduler · kwok
Security   Intel TDX · OAuth 2.1 / OIDC · Kyverno · OPA · cosign · SLSA · OpenSSF model signing
Observability   Prometheus · Grafana · Loki · Tempo · OpenTelemetry · ClickHouse

• Make changes reviewable. Reproducible plans, GitOps reconciliation, and keyless delivery.
• Make failures understandable. Useful telemetry, explicit failure modes, and written runbooks.
• Make correctness repeatable. Regression tests for security fixes, billing paths, and migrations.
• Own the whole path. Design, delivery, operations, and the details between them.

Building AI platforms, developer tools, or secure infrastructure?
sunilmehta.si · Connect on LinkedIn · Connect on X · Email me