Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion container-images/defining-container-images.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ python_version = 3.11

The Python version affects the entire dependency chain. For instance, some packages may not support newer Python versions immediately after release.

To use a later Python version, please use a [Dockerfile](http://localhost:3000/container-images/custom-dockerfiles)
To use a later Python version, please use a [Dockerfile](/container-images/custom-dockerfiles).

<Warning>
Changes to the Python version trigger a full rebuild since they affect both
Expand Down
4 changes: 2 additions & 2 deletions migrations/hugging-face.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -206,13 +206,13 @@ Make sure to replace `[CEREBRIUM_API_KEY]` with your Inference API key, which ca

When migrating, keep the following points in mind:

1. **API structure**: The Cerebrium implementation uses a different API structure compared to Huggingface
1. **API structure**: The Cerebrium implementation uses a different API structure compared to Hugging Face
2. **Authentication**: Ensure you have set up the `HF_AUTH_TOKEN` secret in Cerebrium for authenticating with Hugging Face
3. **Model permissions**: The example uses the Llama 3.1 8B Instruct model. Ensure you have the necessary permissions to use this model
4. **Hardware optimization**: The `cerebrium.toml` file specifies the hardware requirements. Adjust these based on your specific model and performance needs
5. **Dependency management**: Regularly review and update the dependencies listed in `cerebrium.toml` to ensure you're using the latest compatible versions
6. **Scaling configuration**: The example sets up auto-scaling with 0 to 5 replicas and a 30-second cooldown. Monitor your usage patterns and adjust these parameters as needed
7. **Cold starts**: While Cerebrium handles cold starts more gracefully than Huggingface, be aware that the first request after a period of inactivity may still take longer to process
7. **Cold starts**: While Cerebrium handles cold starts more gracefully than Hugging Face, be aware that the first request after a period of inactivity may still take longer to process
8. **Monitoring and logging**: Familiarize yourself with Cerebrium's monitoring and logging capabilities to track your model's performance and usage
9. **Cost management**: Although Cerebrium's pay-per-use model can be more cost-effective, set up proper monitoring and alerts to avoid unexpected costs
10. **Testing**: Thoroughly test your migrated models to ensure they perform as expected on the new platform
Expand Down
4 changes: 2 additions & 2 deletions performance/checkpointing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,12 @@ engine.wake_up()

**Ephemeral filesystem:** Any files written to disk before the checkpoint are not copied to the restored container. Only memory is checkpointed.

**Provider Availability:** Checkpointing is only available on the <b>AWS provider</b>. More coming soon.
**Provider availability:** Checkpointing is only available on the <b>AWS provider</b>. More coming soon.

## Platform-specific recommendations

### vLLM

vLLM checkpointing support is not complete but is still possible. See [vllm-project/vllm#34303](https://github.com/vllm-project/vllm/issues/34303) and related issues.

The larger the size of the memory checkpoint the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/).
The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/).
8 changes: 4 additions & 4 deletions v4/examples/aiVoiceAgents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ You can find the final version of the code [here](https://github.com/CerebriumAI
See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium.

<Note>
You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else
You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium else
you must use their API endpoint below.
</Note>

Expand Down Expand Up @@ -199,7 +199,7 @@ dotenv = "latest"
The Docker base image contains local [Deepgram](https://deepgram.com/) Speech-to-Text (STT) and Text-to-Speech (TTS) models, provided by Daily. Running everything locally instead of over the network achieves low latency.

<Note>
Docker files are not support yet but are rather in the works to be released
Docker files are not supported yet but are rather in the works to be released
soon. This is just a very early preview of how it would work.
</Note>

Expand Down Expand Up @@ -393,8 +393,8 @@ The code runs in a separate execution environment to prevent multiple PipeCat in
<NOTE>
Currently, Cerebrium does not support workloads running longer than 5 minutes
however it is currently being worked on internally and will be released soon.
This means that conversations are limited to a 5 minute window. If this is a
issue and you have a urgent use case, please reach out to
This means that conversations are limited to a 5 minute window. If this is an
issue and you have an urgent use case, please reach out to
[support](mailto:support@cerebrium.ai)
</NOTE>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This tutorial deploys a Vision Language Model (VLM) using SGLang on Cerebrium. A

The example builds an intelligent ad analysis system that evaluates advertisements across multiple dimensions, scoring how the advertisement relates to the business in question and how it performs on the given criteria.

SGLang (Structured Generation Language) differs from other inference frameworks such as vLLM and TensorRT by focusing no structed generation and complex workflows multi-step LLM workflows. SGLang is being used in production by teams at xAI and Deepseek to power their core language model capabilities making it a trusted choice.
SGLang (Structured Generation Language) differs from other inference frameworks such as vLLM and TensorRT by focusing on structured generation and complex multi-step LLM workflows. SGLang is being used in production by teams at xAI and Deepseek to power their core language model capabilities making it a trusted choice.

### SGLang Architecture

Expand Down
4 changes: 2 additions & 2 deletions v4/examples/high-throughput-embeddings.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Deploy a High Throughput Server for Embeddings and Reranking"
sidebarTitle: "High-Throughput Embeddings Server"
description: "Deploy a a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali"
description: "Deploy a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali"
---

This tutorial covers deploying a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap, and colpali using the open-source framework
Expand All @@ -15,7 +15,7 @@ Find the final working version [here](https://github.com/CerebriumAI/examples/tr

### Project Setup

Complete the [quickstart]() to install the CLI and create an account.
Complete the [quickstart](/getting-started/introduction) to install the CLI and create an account.

1. Run the command: `cerebrium init infinity-throughput`

Expand Down
2 changes: 1 addition & 1 deletion v4/examples/openai-compatible-endpoint-vllm.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "OpenAI compatible vLLM endpoint"
description: "Create a OpenAI compatible endpoint using the vLLM framework"
description: "Create an OpenAI compatible endpoint using the vLLM framework"
---

This tutorial creates an OpenAI-compatible endpoint that works with any open-source model. Use existing OpenAI code with Cerebrium serverless functions by changing just two lines of code.
Expand Down
2 changes: 1 addition & 1 deletion v4/examples/realtime-voice-agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Create a Cerebrium account by signing up [here](https://dashboard.cerebrium.ai/r
See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium.

<Note>
You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else
You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium else
you must use their API endpoint below.
</Note>

Expand Down
2 changes: 1 addition & 1 deletion v4/examples/sdxl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/3-sdxl-refiner/predict'
--data '{
"url": "https://huggingface.co/datasets/patrickvonplaten/images/resolve/main/aa_xl/000000009.png",
"prompt": "a photo of an astronaut riding a horse on mars"
}''
}'
```

The endpoint returns results in this format:
Expand Down
2 changes: 1 addition & 1 deletion v4/examples/transcribe-whisper.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ After deployment, make this request:
curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/1-whisper-transcription/predict' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <YOUR TOKEN HERE>' \
--data '{"file_url": "https://your-public-url.com/test.mp3"}''
--data '{"file_url": "https://your-public-url.com/test.mp3"}'
```

The response returns immediately with a 202 status code and a `run_id` — a unique identifier to correlate the result with the initial workload.
Expand Down
2 changes: 1 addition & 1 deletion v4/examples/twilio-voice-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ entrypoint = ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8765"]
healthcheck_endpoint = "/health"
```

You can read more about run custom web servers [here](https://docs.cerebrium.ai/container-images/custom-web-servers).
You can read more about running custom web servers [here](/container-images/custom-web-servers).

### Twilio setup

Expand Down
Loading