diff --git a/container-images/custom-web-servers.mdx b/container-images/custom-web-servers.mdx index 0eab95e0..e0874989 100644 --- a/container-images/custom-web-servers.mdx +++ b/container-images/custom-web-servers.mdx @@ -5,7 +5,7 @@ description: "Run ASGI/WSGI Python apps on Cerebrium" Cerebrium's default runtime covers most app needs. For more control, use ASGI or WSGI servers through the custom runtime feature - enabling custom authentication, dynamic batching, frontend dashboards, public endpoints, and WebSocket connections. -## Setting Up Custom Servers +## Setting up custom servers A basic FastAPI server running as a custom server on Cerebrium: @@ -58,6 +58,6 @@ The configuration requires three key parameters: The [FastAPI Server Example](https://github.com/CerebriumAI/examples) provides a complete implementation. -## Request Headers +## Request headers Custom web servers receive the Cerebrium run ID in the `X-Request-Id` header on every request. This corresponds to the internal `run_id` and is useful for tracking and debugging. diff --git a/container-images/defining-container-images.mdx b/container-images/defining-container-images.mdx index 1e125f15..5bdd9912 100644 --- a/container-images/defining-container-images.mdx +++ b/container-images/defining-container-images.mdx @@ -12,7 +12,7 @@ Unlike traditional Docker or Kubernetes setups with multiple configuration files Python decorators scatter infrastructure settings throughout code files, making changes risky and reviews difficult. TOML centralizes configuration in one place, making it easier to track changes and maintain consistency. Its hierarchical structure maps naturally to app requirements without the accidental complexity of code-based configuration. -### Getting Started +### Getting started Run `cerebrium init` to create a `cerebrium.toml` file in the project root. Edit it to match the app's requirements. @@ -24,7 +24,7 @@ Run `cerebrium init` to create a `cerebrium.toml` file in the project root. Edit section of your `cerebrium.toml` file. -## Hardware Configuration +## Hardware configuration Configure GPU type and memory allocations in the hardware section: @@ -40,7 +40,7 @@ For detailed hardware specifications see the [toml reference](/toml-reference/to ## Dependency management -### Selecting a Python Version +### Selecting a Python version The Python runtime version forms the foundation of every Cerebrium app. Supported versions: 3.10 to 3.13. Specify the version in the deployment section: @@ -58,7 +58,7 @@ To use a later Python version, please use a [Dockerfile](http://localhost:3000/c the base environment and all Python package installations. -### Adding Python Packages +### Adding Python packages Manage Python dependencies directly in TOML or through requirement files: @@ -83,7 +83,7 @@ pip = "requirements.txt" Cerebrium caches pip packages at the node level - including wheel files and compiled binaries - so subsequent builds only install new or updated packages. This significantly reduces build times. -### Adding APT Packages +### Adding APT packages System-level packages (image-processing libraries, audio codecs, etc.) are declared under `[cerebrium.dependencies.apt]`: @@ -103,7 +103,7 @@ apt = "deps_folder/pkglist.txt" Changes to APT packages trigger a full rebuild of the container image, so builds take longer than when modifying Python packages alone. -### Conda Packages +### Conda packages Conda excels at managing complex system-level Python dependencies, particularly for GPU support and scientific computing: @@ -123,11 +123,11 @@ conda = "conda_pkglist.txt" Like APT packages, Conda packages modify system-level components. Changes trigger a full rebuild. Batch Conda dependency updates together to minimize rebuild time. -## Build Commands +## Build commands The build process includes two command types that execute at different stages during container image creation. -### Pre-build Commands +### Pre-build commands Pre-build commands execute at the start of the build process, before dependency installation. Use them to set up the build environment: @@ -143,7 +143,7 @@ pre_build_commands = [ Common uses: installing build tools, configuring system settings, or preparing the environment for subsequent build steps. -### Shell Commands +### Shell commands Shell commands execute after all dependencies install and the application code copies into the container. This later timing ensures access to the complete environment: @@ -159,11 +159,11 @@ shell_commands = [ Use shell commands for tasks that require the fully configured environment — such as compiling code that depends on installed libraries or downloading resources. -## Custom Docker Base Images +## Custom Docker base images The base image determines the OS foundation for the container. The default Debian slim image works for most Python apps; other validated base images support specific requirements. -### Supported Base Images +### Supported base images Supported base image categories include NVIDIA, Ubuntu, and Python images. @@ -182,7 +182,7 @@ docker_base_image_url = "debian:bookworm-slim" # Default minimal image add only essential components as needed. -#### Public Docker Hub Images with Namespaces +#### Public Docker Hub images with namespaces Public Docker Hub images with a namespace (e.g., `bob/infinity`, `huggingface/transformers`) require a local Docker Hub login, even though the image is public. Cerebrium reads `~/.docker/config.json` to authenticate image pulls. @@ -211,7 +211,7 @@ docker_base_image_url = "bob/infinity:latest" with our build system. -#### Public AWS ECR Images +#### Public AWS ECR images Public ECR images from the `public.ecr.aws` registry work without authentication: @@ -222,11 +222,11 @@ docker_base_image_url = "public.ecr.aws/lambda/python:3.11" However, **private ECR images** require authentication. See [Using Private Docker Registries](/container-images/private-docker-registry) for setup instructions. -## Custom Runtimes +## Custom runtimes Cerebrium's default runtime covers most apps. Custom runtimes provide more control, enabling features like custom authentication, dynamic batching, public endpoints, or WebSocket connections. -### Basic Configuration +### Basic configuration Define a custom runtime by adding the `cerebrium.runtime.custom` section to the configuration: @@ -252,7 +252,7 @@ Key parameters: for a detailed implementation of a FastAPI server that uses a custom runtime. -### Self-Contained Servers +### Self-contained servers Custom runtimes also support apps with built-in servers. For example, deploying a VLLM server requires no Python code: @@ -268,7 +268,7 @@ torch = "latest" vllm = "latest" ``` -### Important Notes +### Important notes - Code is mounted in `/cortex` - adjust paths accordingly. - The port in your entrypoint must match the `port` parameter. @@ -283,11 +283,11 @@ Deploy with `cerebrium deploy -y` - the system automatically detects custom runt The build process follows a sequence that transforms source code into a production-ready container image: -### Stage 1: App Upload +### Stage 1: App upload Code is uploaded to Cerebrium, including all source files, configuration, and additional assets needed for the app. -### Stage 2: Image Creation +### Stage 2: Image creation The system creates a container image through the following sequential steps: @@ -298,6 +298,6 @@ The system creates a container image through the following sequential steps: 5. **Python Code Copy**: The app's source code copies into the container, placing it in the correct directory structure. 6. **Shell Commands Execute**: Finally, any build-time shell commands run to complete the image setup. -### Stage 3: Production Image +### Stage 3: Production image The result is a production-ready container image that contains everything needed to run the app. This image serves as a blueprint for creating individual containers when the app receives requests. diff --git a/deployments/multi-region-deployment.mdx b/deployments/multi-region-deployment.mdx index 4de050d9..0318e1a6 100644 --- a/deployments/multi-region-deployment.mdx +++ b/deployments/multi-region-deployment.mdx @@ -12,13 +12,13 @@ Deploy apps globally across multiple continents to reduce latency through co-loc about features/functionality you would like to see -## Why Use Multi-Region Deployment +## Why use multi-region deployment - **Reduced Latency**: Deploy closer to users for faster response times and better experience with real-time applications like voice agents and LLMs - **Data Residency**: Meet data protection requirements by keeping sensitive data within specific geographic regions to comply with regulations like GDPR and CCPA - **High Availability**: Ensure fault tolerance and continuous service through geographic redundancy, disaster recovery, and load balancing across multiple regions -## Available Regions +## Available regions Cerebrium supports deployment across three major continents with the following regions: @@ -42,7 +42,7 @@ Cerebrium supports deployment across three major continents with the following r region not currently listed. -## CLI Configuration +## CLI configuration Configure the CLI to work with different regions in two ways: @@ -68,7 +68,7 @@ cerebrium download results.json -r eu-north-1 `cerebrium region set` is not modified. -## App Deployment +## App deployment Set the deployment region using the `region` parameter in the `[cerebrium.hardware]` section of `cerebrium.toml`: @@ -85,7 +85,7 @@ memory = 8.0 Pricing varies by region based on local infrastructure costs and availability: -## GPU Availability by Region +## GPU availability by region GPU availability and pricing vary across regions due to infrastructure constraints and local demand: diff --git a/endpoints/webhook.mdx b/endpoints/webhook.mdx index c0632f15..2b588cf6 100644 --- a/endpoints/webhook.mdx +++ b/endpoints/webhook.mdx @@ -14,7 +14,7 @@ curl -X POST https://api.cerebrium.ai/v4/p-xxxxxxxx//run?webhookEndpoi The proxy forwards the response body as a POST request to the specified webhook — no code changes required. Ensure the webhook endpoint accepts POST requests. Webhook forwarding works with both Cortex and Custom runtimes, but only for HTTP requests (not WebSockets). -## Retry Behavior +## Retry behavior Webhook requests are sent asynchronously and do not block the function's response. Failed deliveries are retried automatically: @@ -51,7 +51,7 @@ curl -X POST https://webhook.site/'\ --data '{"smile": "wave"}' ``` -## Webhook Signature Verification +## Webhook signature verification Verify that webhooks originate from Cerebrium using signature verification: diff --git a/endpoints/websockets.mdx b/endpoints/websockets.mdx index 0ea371d3..d49197f6 100644 --- a/endpoints/websockets.mdx +++ b/endpoints/websockets.mdx @@ -37,7 +37,7 @@ Test the WebSocket endpoint using websocat, a command-line WebSocket client: websocat wss://api.cerebrium.ai/v4/p-xxxxxxxx// ``` -## Implementing the WebSocket Endpoint +## Implementing the WebSocket endpoint Example WebSocket endpoint using FastAPI: @@ -54,7 +54,7 @@ async def websocket_endpoint(websocket: WebSocket): await websocket.close() ``` -## Additional Info +## Additional info Client-side Implementation: Handle the WebSocket connection properly on the client, including error handling and reconnection logic. diff --git a/hardware/using-gpus.mdx b/hardware/using-gpus.mdx index f95dab0b..967ffbd2 100644 --- a/hardware/using-gpus.mdx +++ b/hardware/using-gpus.mdx @@ -39,7 +39,7 @@ The platform offers GPUs ranging from cost-effective development options to high during application initialization. -## Multi-GPU Configuration +## Multi-GPU configuration Multiple GPUs are configured in the `cerebrium.toml` file: diff --git a/migrations/mystic.mdx b/migrations/mystic.mdx index 66bb56ea..3c167240 100644 --- a/migrations/mystic.mdx +++ b/migrations/mystic.mdx @@ -9,7 +9,7 @@ Mystic AI is sunsetting their services. They were an early pioneer that pushed t It covers converting existing Mystic code (using a stable diffusion example) and configuration to the Cerebrium platform, including deployment optimization for performance and cost efficiency. -## Key Differences +## Key differences Cerebrium helps teams deploy and run models efficiently. The infrastructure is designed for reliable performance: @@ -19,9 +19,9 @@ Cerebrium helps teams deploy and run models efficiently. The infrastructure is d Cerebrium provides precise control over computing resources. Instead of managing entire instances, select the exact CPU, memory, and GPU power needed. Billing is per-second for actual resource usage. Use the [pricing calculator](https://cerebrium.ai/pricing) for cost estimates. -## Migration Process +## Migration process -### 1. Project Setup and Configuration +### 1. Project setup and configuration Install Cerebrium's command-line tool and create the project: @@ -92,7 +92,7 @@ xformers = "latest" ``` -### 2. Code Migration +### 2. Code migration Convert the model implementation. A typical Mystic pipeline: @@ -283,7 +283,7 @@ curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/stable-diffusion/predict The Cerebrium platform provides the tools and support needed for a smooth transition. -## Join the Community +## Join the community Connect with other developers and the Cerebrium team for faster response and issue resolution: diff --git a/other-topics/using-secrets.mdx b/other-topics/using-secrets.mdx index 06924710..1720712f 100644 --- a/other-topics/using-secrets.mdx +++ b/other-topics/using-secrets.mdx @@ -27,7 +27,7 @@ def predict(run_id): `json.loads(os.environ.get("MY_JSON_SECRET"))`. -### Managing Secrets +### Managing secrets Secrets are created, updated, and deleted in your dashboard. @@ -38,7 +38,7 @@ Secrets are created, updated, and deleted in your dashboard. container to restart or deploy your app before the new secret is available. -### Automatic Environment Variables +### Automatic environment variables Cerebrium automatically sets the following environment variables for your app: @@ -49,7 +49,7 @@ Cerebrium automatically sets the following environment variables for your app: The app_id is a composite of PROJECT_ID + '_' + APP_NAME. -### Local Development +### Local development For local development, use an `.env` file. Add the same secrets to the dashboard before deploying. diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index 3fb1fee8..395b6d8d 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -11,7 +11,7 @@ This is useful for both CPU-only and GPU workloads. For CPU applications, checkp For example, ML and LLM frameworks often load large model weights and compile CUDA kernels at container start time, which can take many seconds or minutes. Loading from a checkpoint that already contains this initialized state can skip most of that delay. -Since this feature is still in beta, please report all issues to the team via our [Discord Community](https://discord.gg/ATj6USmeE2) or via [Email](mailto:support@cerebrium.ai). +Since this feature is still in beta, please report all issues to the team via our [Discord community](https://discord.gg/ATj6USmeE2) or via [email](mailto:support@cerebrium.ai). ## How to use @@ -66,7 +66,7 @@ If checkpoint creation succeeds, subsequent containers restore from that snapsho A checkpoint is tightly coupled to a single deployment. To stop restoring from checkpoints, remove the POST request and redeploy the application. -You can find several implementations in our [Examples repository on Github](https://github.com/CerebriumAI/examples). +You can find several implementations in our [examples repository on GitHub](https://github.com/CerebriumAI/examples). ### vLLM example @@ -111,7 +111,7 @@ engine.wake_up() **Ephemeral filesystem:** Any files written to disk before the checkpoint are not copied to the restored container. Only memory is checkpointed. -**Provider Availability:** Checkpointing is only available on the AWS provider. More coming soon. +**Provider availability:** Checkpointing is only available on the **AWS provider**. More coming soon. ## Platform-specific recommendations @@ -119,4 +119,4 @@ engine.wake_up() vLLM checkpointing support is not complete but is still possible. See [vllm-project/vllm#34303](https://github.com/vllm-project/vllm/issues/34303) and related issues. -The larger the size of the memory checkpoint the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). +The larger the memory checkpoint, the slower the restore. To reduce snapshot size and improve startup times, drop the KV cache before checkpointing and recreate it after restore. vLLM has built-in functionality for this as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). diff --git a/scaling/scaling-apps.mdx b/scaling/scaling-apps.mdx index 0fdc1dff..a31e6783 100644 --- a/scaling/scaling-apps.mdx +++ b/scaling/scaling-apps.mdx @@ -5,11 +5,11 @@ description: "Learn to optimise for cost and performance by scaling out apps" Cerebrium's scaling system automatically manages computing resources to match app demand, from single requests to multiple simultaneous requests, optimizing for both performance and cost. -## How Autoscaling Works +## How autoscaling works The scaling system monitors a configurable metric - such as concurrency utilization, requests per second, CPU usage, or memory usage - and compares it against a target threshold. When the metric exceeds the target, new instances start within seconds. See [Scaling Metrics](#using-scaling-metrics) for details on each option. -## Scaling Configuration +## Scaling configuration The `cerebrium.toml` file controls scaling behavior through several key parameters: @@ -21,19 +21,19 @@ cooldown = 60 # Cooldown period in seconds replica_concurrency = 1 # The maximum number of requests each replica of an app can accept ``` -### Minimum Instances +### Minimum instances The `min_replicas` parameter defines how many instances remain active at all times. Setting this to 1 or higher maintains warm instances for immediate response, eliminating cold starts but increasing costs. Use this for apps requiring consistent response times or specific SLA guarantees. -### Maximum Instances +### Maximum instances The `max_replicas` parameter sets an upper limit on concurrent instances, controlling costs and protecting backend systems. When traffic increases, new instances start automatically up to this configured maximum. -### Cooldown Period +### Cooldown period The `cooldown` parameter specifies the time window (in seconds) that must pass at reduced concurrency before an instance scales down. This prevents premature scale-down during brief traffic dips that might be followed by more requests. A longer cooldown period helps handle bursty traffic patterns but increases instance running time and cost. -### Replica Concurrency +### Replica concurrency The number of requests an app instance can handle concurrently is dictated by the `replica_concurrency` parameter. This is a hard limit, and an individual replica will not accept more than this limit at a time. By default, once this concurrency limit is reached on an instance and there are still requests to be processed in-flight, @@ -47,11 +47,11 @@ _3_ requests in flight with no replicas currently available, Cerebrium will scal controlled within the application through batching. -## Processing Multiple Requests +## Processing multiple requests Apps can process multiple requests simultaneously through batching and concurrency. Cerebrium supports frameworks with built-in batching and enables custom implementations through the [custom runtime](container-images/defining-container-images#custom-runtimes) feature. See the [Batching & Concurrency Guide](/scaling/batching-concurrency) for details. -## Instance Management +## Instance management Cerebrium automatically restarts failed instances, starts new instances to maintain capacity, and monitors instance health continuously. @@ -86,7 +86,7 @@ Performance metrics available through the dashboard help monitor scaling behavio - Cold start frequency - Resource usage patterns -## Using Scaling Metrics +## Using scaling metrics Cerebrium supports multiple scaling metrics beyond the default `replica_concurrency`. Four scaling metrics are available: @@ -108,7 +108,7 @@ scaling_metric = "concurrency_utilization" scaling_target = 100 ``` -### Concurrency Utilization +### Concurrency utilization `concurrency_utilization` is the default scaling metric, with a default target of _100%_. This metric maintains a maximum percentage of `replica_concurrency` averaged across all instances. @@ -116,19 +116,19 @@ For example, with `replica_concurrency=1` and `scaling_target=70`, Cerebrium mai With `replica_concurrency=200` and `scaling_target=80`, Cerebrium maintains _160_ requests per instance and scales out once that target is exceeded. -### Requests per Second +### Requests per second `requests_per_second` maintains a maximum application throughput measured in requests per second, averaged over all instances. This metric is more effective than `concurrency_utilization` when application throughput has been benchmarked. It does not enforce concurrency limits, so it is not recommended for most GPU applications. For example, `scaling_target=5` maintains a 5 requests/s average across all instances. -### CPU Utilization +### CPU utilization `cpu_utilization` scales based on maximum CPU percentage utilization averaged over all instances, relative to the `cerebrium.hardware.cpu` value. For example, with `cpu=2` and `scaling_target=80`, Cerebrium maintains _80%_ CPU utilization (1.6 CPUs) per instance. This metric requires `min_replicas=1` since scaling relative to 0 CPU units is undefined. -### Memory Utilization +### Memory utilization `memory_utilization` scales based on maximum RAM percentage utilization averaged over all instances, relative to `cerebrium.hardware.memory`. This refers to RAM, **not** GPU VRAM. For example, with `memory=10` and `scaling_target=80`, Cerebrium maintains _80%_ memory utilization (8GB) per instance. This metric requires `min_replicas=1` since scaling relative to 0GB of memory is undefined. -## Keeping a Scaling Buffer +## Keeping a scaling buffer For apps with long startup times or predictable traffic, a replica buffer maintains consistent excess capacity above what the scaling metric suggests. The `scaling_buffer` option adds a fixed number of extra replicas to the autoscaler's recommendation. This is available with the following scaling metrics: @@ -154,7 +154,7 @@ With `concurrency_utilization` at `100` and `replica_concurrency=1`, receiving 1 The buffer adds a static number of replicas on top of the autoscaler's recommendation. After the request completes, the `cooldown` period applies and the replica count scales back to the **1 replica** baseline. -## Evaluation Interval +## Evaluation interval Requires CLI version 2.1.5 or higher. @@ -173,7 +173,7 @@ A shorter interval makes the autoscaler more responsive to traffic spikes but ma interval (e.g., 60 seconds) reduces unnecessary scaling churn. -## Load Balancing +## Load balancing Requires CLI version 2.1.5 or higher. @@ -186,7 +186,7 @@ load_balancing_algorithm = "min-connections" # Explicitly set load balancing al **Default behavior**: When `load_balancing_algorithm` is not set, the system uses `first-available` for `replica_concurrency <= 3` (typical for GPU workloads) and `round-robin` for higher concurrency. -### Available Algorithms +### Available algorithms #### round-robin @@ -240,7 +240,7 @@ Implements the "Power of Two Choices" algorithm: randomly samples two replicas a **Note**: Uses weight-based tracking rather than reservation-based concurrency limiting, making it suitable for unlimited concurrency scenarios. -### Choosing an Algorithm +### Choosing an algorithm | Scenario | Recommended | Reason | | ---------------------------------------- | --------------------------- | -------------------------------------------------- | @@ -250,7 +250,7 @@ Implements the "Power of Two Choices" algorithm: randomly samples two replicas a | Uniform request times, even distribution | `round-robin` | Predictable rotation, no hot spots over time | | Latency-sensitive with variable load | `min-connections` | Minimizes p90/p99 by routing to least busy replica | -## Compute Tier +## Compute tier Requires CLI version 2.1.6 or higher. @@ -261,7 +261,7 @@ The `compute_tier` parameter controls whether your replicas are scheduled on spo compute_tier = "protected" # Use on-demand instances for higher availability ``` -### Available Tiers +### Available tiers | Tier | Description | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/storage/managing-files.mdx b/storage/managing-files.mdx index ea127dba..1b55dd78 100644 --- a/storage/managing-files.mdx +++ b/storage/managing-files.mdx @@ -10,7 +10,7 @@ Cerebrium offers file management through a 50GB persistent volume that's availab manage these files. -## Including Files in Deployments +## Including files in deployments The `cerebrium.toml` configuration file controls which files become part of the app: @@ -30,7 +30,7 @@ exclude = [ Files included in deployments must be under 2GB each, with deployments working best for files under 1GB. Larger files should use persistent storage instead. -## Managing Persistent Storage +## Managing persistent storage The CLI provides four commands for working with persistent storage. @@ -93,7 +93,7 @@ cerebrium cp model.bin -r eu-north-1 cerebrium download sub_folder/file_name.txt ``` -## Using Stored Files +## Using stored files Access files in persistent storage at runtime: @@ -128,7 +128,7 @@ if os.path.exists("/persistent-storage/models/mymodel.pt"): at runtime. -## Increasing Storage Capacity +## Increasing storage capacity The default 50GB persistent volume can be increased up to 1TB through self-service. The first 100GB is free, then $0.05 per GB per month. @@ -156,7 +156,7 @@ curl -X POST "https://api.cerebrium.ai/v4/projects/{project_id}/volumes/{volume_ [email](mailto:support@cerebrium.ai). -### Storage Quota Errors +### Storage quota errors If you hit your storage quota, you'll see errors like `No space left on device` when trying to download or write files. To resolve this: diff --git a/toml-reference/toml-reference.mdx b/toml-reference/toml-reference.mdx index ec9c8c23..283ece5f 100644 --- a/toml-reference/toml-reference.mdx +++ b/toml-reference/toml-reference.mdx @@ -11,7 +11,7 @@ The configuration is organized into the following main sections: - **[cerebrium.scaling]** Auto-scaling behavior and replica management - **[cerebrium.dependencies]** Package management for Python (pip), system (apt), and Conda dependencies -## Deployment Configuration +## Deployment configuration The `[cerebrium.deployment]` section defines core deployment settings. @@ -33,7 +33,7 @@ The `[cerebrium.deployment]` section defines core deployment settings. they affect the base environment. -### UV Package Manager +### UV package manager UV is a fast Python package installer written in Rust that significantly speeds up deployment times. When enabled, UV replaces pip for installing Python dependencies. @@ -53,7 +53,7 @@ UV typically installs packages 10-100x faster than pip, especially beneficial fo use_uv = true ``` -### Monitoring UV Usage +### Monitoring UV usage Check your build logs for these indicators: @@ -65,7 +65,7 @@ Check your build logs for these indicators: failures, such as legacy packages with non-standard metadata. -### Deploying with UV Lock Files +### Deploying with UV lock files read only if you're using `pyproject.toml` and `uv.lock` @@ -89,7 +89,7 @@ Include in your deployment: - Ensure requirements.txt is in your project directory - Deploy with UV enabled -## Runtime Configuration +## Runtime configuration The `[cerebrium.runtime.custom]` section configures custom web servers and runtime behavior. @@ -106,7 +106,7 @@ The `[cerebrium.runtime.custom]` section configures custom web servers and runti `https://api.cerebrium.ai/v4/p-xxxxxxxx/your-app-name/your/endpoint` -## Hardware Configuration +## Hardware configuration The `[cerebrium.hardware]` section defines compute resources. @@ -124,7 +124,7 @@ The `[cerebrium.hardware]` section defines compute resources. workload. -## Scaling Configuration +## Scaling configuration The `[cerebrium.scaling]` section controls auto-scaling behavior. @@ -165,7 +165,7 @@ For example, with `min_replicas=0` and `scaling_buffer=3`, the system will maint ## Dependencies -### Pip Dependencies +### Pip dependencies The `[cerebrium.dependencies.pip]` section lists Python package requirements. @@ -176,7 +176,7 @@ numpy = "latest" # Latest version pandas = ">=1.5.0" # Minimum version ``` -### APT Dependencies +### APT dependencies The `[cerebrium.dependencies.apt]` section specifies system packages. @@ -186,7 +186,7 @@ ffmpeg = "latest" libopenblas-base = "latest" ``` -### Conda Dependencies +### Conda dependencies The `[cerebrium.dependencies.conda]` section manages Conda packages. @@ -196,7 +196,7 @@ cuda = ">=11.7" cudatoolkit = "11.7" ``` -### Dependency Files +### Dependency files The `[cerebrium.dependencies.paths]` section allows using requirement files. @@ -207,7 +207,7 @@ apt = "pkglist.txt" conda = "conda_pkglist.txt" ``` -## Complete Example +## Complete example ```toml [cerebrium.deployment] diff --git a/v4/examples/aiVoiceAgents.mdx b/v4/examples/aiVoiceAgents.mdx index 32e7bdd5..ac2e10ad 100644 --- a/v4/examples/aiVoiceAgents.mdx +++ b/v4/examples/aiVoiceAgents.mdx @@ -28,7 +28,7 @@ See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram you must use their API endpoint below. -### LLM Deployment +### LLM deployment The LLM is an OpenAI-compatible Llama-3 endpoint using the vLLM framework. For low TTFT, a quantized version is used (RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8). @@ -413,7 +413,7 @@ def start_bot(room_url: str, token: str = None): The result is a fully functioning AI bot that interacts with users through speech in ~500ms. The next section creates a user interface. -## Creating Meeting Room +## Creating a meeting room Cerebrium runs any Python code, not just AI workloads. The following two functions use the Daily REST API to create a room and temporary token, both valid for 5 minutes. diff --git a/v4/examples/comfyUI.mdx b/v4/examples/comfyUI.mdx index 14e65596..c9e8d066 100644 --- a/v4/examples/comfyUI.mdx +++ b/v4/examples/comfyUI.mdx @@ -12,11 +12,11 @@ ComfyUI is a popular no-code interface for building complex stable diffusion wor Production-scale deployment guidance for ComfyUI is limited. This tutorial covers deploying ComfyUI pipelines on Cerebrium as autoscaling API endpoints with pay-as-you-go compute. Find the full example code [here](https://github.com/CerebriumAI/examples/tree/master/7-image-and-video/1-comfyui). -### Creating your Comfy UI workflow locally +### Creating your ComfyUI workflow locally Create the workflow locally or using a rented GPU from [Lambda Labs](https://lambdalabs.com/). Ensure [ComfyUI is installed](https://github.com/comfyanonymous/ComfyUI#installing) in the local environment. -This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. Skip to "Export ComfyUI Workflow" if a workflow already exists. +This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. Skip to "Export the ComfyUI workflow" if a workflow already exists. 1. Create the Cerebrium project: `cerebrium init 1-comfyui` 2. Clone the ComfyUI GitHub project inside the project directory: `git clone https://github.com/comfyanonymous/ComfyUI` @@ -29,7 +29,7 @@ This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. The default ComfyUI workflow interface appears in this view. Use this locally running instance to build the image generation pipeline. -### Export ComfyUI Workflow +### Export the ComfyUI workflow The example GitHub repository contains a workflow.json file. Click the “Load” button on the right to load the workflow. It should appear populated @@ -43,7 +43,7 @@ To export the workflow in API format, click the gear icon (settings) in the top- A button appears in the right hover panel labeled “Save (API format)”. Use it to save the workflow as “workflow_api.json” -### ComfyUI Application +### ComfyUI application The main code in `main.py` uses the exported ComfyUI API workflow to create an endpoint. The code: @@ -256,7 +256,7 @@ def shutdown_event(): terminate_process() ``` -### Deploy ComfyUI Application +### Deploy the ComfyUI application Before deploying, define the deployment configuration in `cerebrium.toml`. This specifies dependencies, hardware, scaling, and any pre-run scripts: diff --git a/v4/examples/gpt-oss.mdx b/v4/examples/gpt-oss.mdx index 3d535b01..f0088e8f 100644 --- a/v4/examples/gpt-oss.mdx +++ b/v4/examples/gpt-oss.mdx @@ -3,9 +3,9 @@ title: "Serving GPT-OSS with vLLM" description: "Deploy OpenAI's Latest Open Source Model" --- -GPT recently released GPT-OSS ([gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) and [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)) two state-of-the-art open-weight language models that deliver strong real-world performance at low cost. Available under the flexible Apache 2.0 license, these models outperform similarly sized open models on reasoning tasks, demonstrate strong tool use capabilities, and are optimized for efficient deployment on consumer hardware. +GPT recently released GPT-OSS ([gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) and [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)): two state-of-the-art open-weight language models that deliver strong real-world performance at low cost. The models are available under the flexible Apache 2.0 license. They outperform similarly sized open models on reasoning tasks, demonstrate strong tool use capabilities, and are optimized for efficient deployment on consumer hardware. -## What Makes GPT-OSS Special? +## What makes GPT-OSS special? GPT-OSS introduces capabilities that set it apart from other open-source LLMs: @@ -15,8 +15,8 @@ GPT-OSS introduces capabilities that set it apart from other open-source LLMs: - **Harmony Response Format**: Built-in support for structured outputs like chain-of-thought reasoning and tool use. See examples from OpenAI [here](https://cookbook.openai.com/articles/openai-harmony) - Please note that in vLLM, you can only run it on NVIDIA H100, H200, B200 as - well as MI300x, MI325x, MI355x and Radeon AI PRO R9700 as of 6th August 2025 + As of 6th August 2025, vLLM only supports running this model on NVIDIA H100, + H200, and B200, as well as MI300x, MI325x, MI355x, and Radeon AI PRO R9700. This tutorial covers the simplest variation of deploying this model using `vllm serve`. For more control, see the [OpenAI compatible endpoint with vLLM guide](/v4/examples/openai-compatible-endpoint-vllm). diff --git a/v4/examples/high-throughput-embeddings.mdx b/v4/examples/high-throughput-embeddings.mdx index 6e4ffb30..288d340b 100644 --- a/v4/examples/high-throughput-embeddings.mdx +++ b/v4/examples/high-throughput-embeddings.mdx @@ -1,7 +1,7 @@ --- title: "Deploy a High Throughput Server for Embeddings and Reranking" sidebarTitle: "High-Throughput Embeddings Server" -description: "Deploy a a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" +description: "Deploy a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" --- This tutorial covers deploying a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap, and colpali using the open-source framework diff --git a/v4/examples/mistral-vllm.mdx b/v4/examples/mistral-vllm.mdx index 293297c8..ce5ba4fa 100644 --- a/v4/examples/mistral-vllm.mdx +++ b/v4/examples/mistral-vllm.mdx @@ -51,9 +51,9 @@ class Item(BaseModel): Pydantic handles data validation. The `prompt` parameter is required; others are optional with default values. A missing `prompt` triggers an automatic error message. -## vLLM Implementation +## vLLM implementation -## Model Setup +### Model setup ```python import os diff --git a/v4/examples/openai-compatible-endpoint-vllm.mdx b/v4/examples/openai-compatible-endpoint-vllm.mdx index 8c0bb58a..d847ca62 100644 --- a/v4/examples/openai-compatible-endpoint-vllm.mdx +++ b/v4/examples/openai-compatible-endpoint-vllm.mdx @@ -106,7 +106,7 @@ The function: - Streams results when `stream=True` using async functionality - Returns the complete result at the end if streaming is disabled -## Deploy & Inference +## Deploy and inference To deploy the model use the following command: diff --git a/v4/examples/realtime-voice-agents.mdx b/v4/examples/realtime-voice-agents.mdx index 493de3fa..ca8d6e60 100644 --- a/v4/examples/realtime-voice-agents.mdx +++ b/v4/examples/realtime-voice-agents.mdx @@ -8,7 +8,7 @@ This tutorial creates a real-time voice agent that responds to queries via speec The app uses [PipeCat](https://www.pipecat.ai/), a framework that handles component integration, user interruptions, and audio data processing. The example joins a meeting room with a voice agent using [Daily](https://daily.co) (PipeCat's creators) and deploys on Cerebrium for scaling. -The application has 3–4 parts: +The application has three parts: - A Pipecat agent that acts as the orchestrator - A Deepgram TTS/STT service (requires a Deepgram Enterprise account) @@ -27,11 +27,11 @@ Create a Cerebrium account by signing up [here](https://dashboard.cerebrium.ai/r See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium. - You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else - you must use their API endpoint below. + You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium. + Otherwise, use their API endpoint below. -### LLM Deployment +### LLM deployment The LLM is an OpenAI-compatible Llama-3 endpoint using the vLLM framework. For low TTFT, a quantized version is used (RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8). diff --git a/v4/examples/streaming-falcon-7B.mdx b/v4/examples/streaming-falcon-7B.mdx index b411a07a..d6f7af9a 100644 --- a/v4/examples/streaming-falcon-7B.mdx +++ b/v4/examples/streaming-falcon-7B.mdx @@ -52,9 +52,9 @@ class Item(BaseModel): Pydantic handles data validation. The `prompt` parameter is required; others are optional with default values. A missing `prompt` triggers an automatic error message. -## Falcon Implementation +## Falcon implementation -## Model Setup +### Model setup ```python from transformers import ( @@ -80,7 +80,7 @@ model = AutoModelForCausalLM.from_pretrained( The tokenizer and model instantiate outside the `predict` function, ensuring model weights load only once at startup. -## Streaming Implementation +### Streaming implementation The `stream` function handles streaming results from the endpoint: diff --git a/v4/examples/transcribe-whisper.mdx b/v4/examples/transcribe-whisper.mdx index 6b0a230b..bc9e8f13 100644 --- a/v4/examples/transcribe-whisper.mdx +++ b/v4/examples/transcribe-whisper.mdx @@ -70,7 +70,7 @@ Pydantic handles data validation. While `audio` and `file_url` are optional para Note: Cerebrium has a 3-minute timeout for each inference request. For long audio files (2+ hours) that take several minutes to process, use a `webhook_endpoint` — a URL where Cerebrium sends a POST request with the function's results. -## Setup Model and inference +## Set up model and inference Import the required packages and load the Whisper model. The model downloads during initial deployment and is automatically cached in persistent storage for subsequent use. Loading the model outside the `predict` function ensures this code only runs on cold start (startup). For warm containers, only the `predict` function executes for inference. diff --git a/v4/examples/twilio-voice-agent.mdx b/v4/examples/twilio-voice-agent.mdx index 3f12338b..a57d5929 100644 --- a/v4/examples/twilio-voice-agent.mdx +++ b/v4/examples/twilio-voice-agent.mdx @@ -119,7 +119,7 @@ Purchase a local number (not toll-free) from the [phone numbers page](https://co Save the changes and proceed to set up the AI agent. -### AI Agent Setup +### AI agent setup Create `bot.py` to set up the AI agent using PipeCat for component integration, interruption handling, and audio processing: @@ -248,7 +248,7 @@ A successful deployment looks like this: Test by calling the Twilio number — the agent responds automatically. -### Scaling Pipecat +### Scaling PipeCat For scaling PipeCat on Cerebrium: