HeteroCloud Flash

HeteroCloud Flash is the isolated container service for HeteroCloud. It is managed from the same HeteroCloud organization and project model as Flow, but runs arbitrary non-privileged TCP and UDP workloads instead of RTC rooms.

Flash keeps the customer API separate from Kubernetes:

HeteroCloud console / API / flashctl
             |
             | signed provider/v1 command
             v
HeteroCloud Flash provider API
             |
             v
FlashService CRD -> Deployment(runtimeClassName=gvisor|nvidia) -> Service
                                                        -> HeteroNetwork LB

Properties

• CPU workloads run with the gvisor Kubernetes RuntimeClass. A spec with gpu_type runs with the nvidia RuntimeClass on a node that passed the cluster GPU smoke test. The provider chooses the runtime from the requested hardware and customers cannot set an arbitrary runtime.
• A VM may select one accessible GPU type. Flash queues the request and chooses the node and logical slot; Kubernetes and the NVIDIA device plugin grant the final exclusive card. See GPU inventory and scheduling.
• TCP and UDP ports use one typed model and support internal ClusterIP or the HeteroNetwork heteronetwork.io/public LoadBalancer class.
• Public forwarded mode distributes traffic to Pods throughout the cluster; direct mode uses externalTrafficPolicy: Local and public-node placement.
• Privileged containers, host networking, host mounts, added Linux capabilities, and service-account token mounts are not generated by the operator.
• Every workload is ingress- and egress-isolated. Only declared inbound ports are exposed; outbound DNS remains available while Pod, Service, LAN, HeteroNetwork, link-local, metadata, and operator-configured infrastructure ranges are denied by default. Tenant-to-tenant traffic requires an explicit same-organization grant.
• Workloads run as container root inside their selected runtime so Web Shell users can use sudo when the image provides it. RuntimeDefault seccomp and the absence of host mounts or Kubernetes credentials keep container root separate from host root; CPU workloads additionally use the runsc sandbox.
• Provider commands use 60-second Ed25519 JWTs, exact issuer/audience/action checks, opaque tenant identifiers, monotonic generations, and deterministic operation IDs.
• The Web console can open /bin/sh in an owned, ready workload Pod. Exec uses a separate IAM action, never exposes Kubernetes credentials, limits each session to 30 minutes, and caps concurrent sessions per provider replica.
• HeteroCloud marks an instance ready only after the requested replicas and its routable endpoint are ready. Terminal Pod startup failures are reported as service errors instead of remaining in provisioning indefinitely. Transient provisioning returns 503 Retry-After to the transactional Outbox and is retried without losing desired state.
• Before creating a Deployment, the controller resolves the OCI image, charges its config and compressed layer bytes against the configured disk limit, and pins the workload to the inspected digest. An image that fills or exceeds the limit is rejected without starting a container. When persistent storage is enabled, the remaining budget is split between a restart-safe /root volume and a bounded writable root filesystem. The volume uses ReadWriteMany, so a rolling update starts its replacement before retiring the current Pod.

Flash spec

Optional autoscaling and domain endpoints are described in Autoscaling and domain endpoints.

{
  "region": "heteronet-global",
  "image": "ghcr.io/ipa-cyberlab/ipa-rs-heterocloud-flash:0.1.23",
  "replicas": 1,
  "cpu_millis": 250,
  "memory_mib": 128,
  "gpu_type": "nvidia-geforce-gtx-1080-ti",
  "ephemeral_storage_gib": 10,
  "ports": [
    {
      "name": "game-udp",
      "protocol": "udp",
      "container_port": 7777
    }
  ],
  "exposure": {
    "type": "public",
    "traffic_mode": "forwarded",
    "allowed_source_cidrs": ["203.0.113.0/24"],
    "denied_source_cidrs": ["203.0.113.128/25"]
  },
  "egress": {
    "mode": "restricted",
    "allow_same_organization": false,
    "allowed_destination_cidrs": ["198.51.100.10/32"],
    "denied_destination_cidrs": []
  },
  "env": {"FLASH_ECHO_LISTEN": "0.0.0.0:7777"},
  "command": ["/usr/local/bin/flash-udp-echo"],
  "args": [],
  "metadata": {}
}

env is durable management-plane data and is intended only for non-secret configuration. Application credentials must not be placed in a Flash spec. The management API assigns each service port from the cluster range. Ports can be added or removed by updating the service; removing every endpoint deletes the Kubernetes Service while the workload remains running, and an unchanged protocol/name pair keeps its assigned public port. Source entries accept individual IPv4/IPv6 addresses or CIDRs. An empty allow list permits publicly routable sources and deny entries always take precedence; private cluster and HeteroNetwork sources remain isolated. Egress mode disabled permits DNS only, restricted permits only the listed public destination CIDRs, and internet permits public Internet destinations. Protected infrastructure CIDRs cannot be granted through the tenant spec. Per-VM and organization-wide resource limits are managed by the HeteroCloud owner quota policy. The OCI image counts toward each VM's configured disk limit.

Flash also meters weekly allocated runtime for CPU, memory, and GPU. CPU is recorded as cpu_millis × ready replicas × seconds, memory as memory_mib × ready replicas × seconds, and GPU as GPU count × ready replicas × seconds. A service at scale zero consumes no runtime quota. Counters reset every Monday at 00:00 UTC; reaching the shared organization CPU or memory limit suspends all of that organization's Flash workloads, while the GPU limit suspends its GPU workloads. Durable FlashUsageRecord resources retain the current week's counters when a service is deleted.

Production clusters should enable restart-safe Web Shell homes and select a CSI storage class that supports ReadWriteMany:

persistence:
  enabled: true
  storageClass: longhorn-static

Files below /root then survive container restarts, Pod replacement, and node rescheduling. Other image filesystem changes remain ephemeral. Web Shell starts in /root and uses /root/.local for user-installed CLI tools.

Platform operators can attach an additional pre-approved PVC to one service without exposing Kubernetes claim selection through the tenant API. The claim is declared with adminVolumeClaims, then assigned by service instance ID with adminVolumeMounts. Credentials remain in the CSI driver Secret and are never added to the workload environment. This is intended for managed object-storage mounts such as a Syouyu-backed S3 CSI volume.

CLI

Create a HeteroCloud API key, store it in a mode-0600 file, and run:

flashctl \
  --organization-id "$ORGANIZATION_ID" \
  --token-file ~/.config/heterocloud/api-token \
  create \
  --project-id "$PROJECT_ID" \
  --name udp-echo \
  --spec examples/udp-echo.json

list, get, update, and delete use the same management API as the Web console.

Development

cargo test --all-targets
cargo clippy --all-targets -- -D warnings
helm lint deploy/helm/heterocloud-flash \
  --set-string providerAuth.publicKeysJson='{}'

Generate the checked-in CRD after changing Rust schema types:

./scripts/generate-crd.sh

See gVisor runtime setup and UDP E2E verification.