diff --git a/container-images/defining-container-images.mdx b/container-images/defining-container-images.mdx index 1e125f15..194191d6 100644 --- a/container-images/defining-container-images.mdx +++ b/container-images/defining-container-images.mdx @@ -51,7 +51,7 @@ python_version = 3.11 The Python version affects the entire dependency chain. For instance, some packages may not support newer Python versions immediately after release. -To use a later Python version, please use a [Dockerfile](http://localhost:3000/container-images/custom-dockerfiles) +To use a later Python version, please use a [Dockerfile](/container-images/custom-dockerfiles). Changes to the Python version trigger a full rebuild since they affect both diff --git a/migrations/hugging-face.mdx b/migrations/hugging-face.mdx index e28ba5b2..3e861f4b 100644 --- a/migrations/hugging-face.mdx +++ b/migrations/hugging-face.mdx @@ -206,13 +206,13 @@ Make sure to replace `[CEREBRIUM_API_KEY]` with your Inference API key, which ca When migrating, keep the following points in mind: -1. **API structure**: The Cerebrium implementation uses a different API structure compared to Huggingface +1. **API structure**: The Cerebrium implementation uses a different API structure compared to Hugging Face 2. **Authentication**: Ensure you have set up the `HF_AUTH_TOKEN` secret in Cerebrium for authenticating with Hugging Face 3. **Model permissions**: The example uses the Llama 3.1 8B Instruct model. Ensure you have the necessary permissions to use this model 4. **Hardware optimization**: The `cerebrium.toml` file specifies the hardware requirements. Adjust these based on your specific model and performance needs 5. **Dependency management**: Regularly review and update the dependencies listed in `cerebrium.toml` to ensure you're using the latest compatible versions 6. **Scaling configuration**: The example sets up auto-scaling with 0 to 5 replicas and a 30-second cooldown. Monitor your usage patterns and adjust these parameters as needed -7. **Cold starts**: While Cerebrium handles cold starts more gracefully than Huggingface, be aware that the first request after a period of inactivity may still take longer to process +7. **Cold starts**: While Cerebrium handles cold starts more gracefully than Hugging Face, be aware that the first request after a period of inactivity may still take longer to process 8. **Monitoring and logging**: Familiarize yourself with Cerebrium's monitoring and logging capabilities to track your model's performance and usage 9. **Cost management**: Although Cerebrium's pay-per-use model can be more cost-effective, set up proper monitoring and alerts to avoid unexpected costs 10. **Testing**: Thoroughly test your migrated models to ensure they perform as expected on the new platform diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index 3fb1fee8..e0fa7da8 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -111,7 +111,7 @@ engine.wake_up() **Ephemeral filesystem:** Any files written to disk before the checkpoint are not copied to the restored container. Only memory is checkpointed. -**Provider Availability:** Checkpointing is only available on the AWS provider. More coming soon. +**Provider availability:** Checkpointing is only available on the AWS provider. More coming soon. ## Platform-specific recommendations @@ -119,4 +119,4 @@ engine.wake_up() vLLM checkpointing support is not complete but is still possible. See [vllm-project/vllm#34303](https://github.com/vllm-project/vllm/issues/34303) and related issues. -The larger the size of the memory checkpoint the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). +The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). diff --git a/v4/examples/aiVoiceAgents.mdx b/v4/examples/aiVoiceAgents.mdx index 32e7bdd5..19e03704 100644 --- a/v4/examples/aiVoiceAgents.mdx +++ b/v4/examples/aiVoiceAgents.mdx @@ -24,7 +24,7 @@ You can find the final version of the code [here](https://github.com/CerebriumAI See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium. - You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else + You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium else you must use their API endpoint below. @@ -199,7 +199,7 @@ dotenv = "latest" The Docker base image contains local [Deepgram](https://deepgram.com/) Speech-to-Text (STT) and Text-to-Speech (TTS) models, provided by Daily. Running everything locally instead of over the network achieves low latency. - Docker files are not support yet but are rather in the works to be released + Docker files are not supported yet but are rather in the works to be released soon. This is just a very early preview of how it would work. @@ -393,8 +393,8 @@ The code runs in a separate execution environment to prevent multiple PipeCat in Currently, Cerebrium does not support workloads running longer than 5 minutes however it is currently being worked on internally and will be released soon. - This means that conversations are limited to a 5 minute window. If this is a - issue and you have a urgent use case, please reach out to + This means that conversations are limited to a 5 minute window. If this is an + issue and you have an urgent use case, please reach out to [support](mailto:support@cerebrium.ai) diff --git a/v4/examples/deploy-a-vision-language-model-with-sglang.mdx b/v4/examples/deploy-a-vision-language-model-with-sglang.mdx index e28dc4eb..336ac68f 100644 --- a/v4/examples/deploy-a-vision-language-model-with-sglang.mdx +++ b/v4/examples/deploy-a-vision-language-model-with-sglang.mdx @@ -7,7 +7,7 @@ This tutorial deploys a Vision Language Model (VLM) using SGLang on Cerebrium. A The example builds an intelligent ad analysis system that evaluates advertisements across multiple dimensions, scoring how the advertisement relates to the business in question and how it performs on the given criteria. -SGLang (Structured Generation Language) differs from other inference frameworks such as vLLM and TensorRT by focusing no structed generation and complex workflows multi-step LLM workflows. SGLang is being used in production by teams at xAI and Deepseek to power their core language model capabilities making it a trusted choice. +SGLang (Structured Generation Language) differs from other inference frameworks such as vLLM and TensorRT by focusing on structured generation and complex multi-step LLM workflows. SGLang is being used in production by teams at xAI and Deepseek to power their core language model capabilities making it a trusted choice. ### SGLang Architecture diff --git a/v4/examples/high-throughput-embeddings.mdx b/v4/examples/high-throughput-embeddings.mdx index 6e4ffb30..88cc713f 100644 --- a/v4/examples/high-throughput-embeddings.mdx +++ b/v4/examples/high-throughput-embeddings.mdx @@ -1,7 +1,7 @@ --- title: "Deploy a High Throughput Server for Embeddings and Reranking" sidebarTitle: "High-Throughput Embeddings Server" -description: "Deploy a a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" +description: "Deploy a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" --- This tutorial covers deploying a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap, and colpali using the open-source framework @@ -15,7 +15,7 @@ Find the final working version [here](https://github.com/CerebriumAI/examples/tr ### Project Setup -Complete the [quickstart]() to install the CLI and create an account. +Complete the [quickstart](/getting-started/introduction) to install the CLI and create an account. 1. Run the command: `cerebrium init infinity-throughput` diff --git a/v4/examples/openai-compatible-endpoint-vllm.mdx b/v4/examples/openai-compatible-endpoint-vllm.mdx index 8c0bb58a..2b689807 100644 --- a/v4/examples/openai-compatible-endpoint-vllm.mdx +++ b/v4/examples/openai-compatible-endpoint-vllm.mdx @@ -1,6 +1,6 @@ --- title: "OpenAI compatible vLLM endpoint" -description: "Create a OpenAI compatible endpoint using the vLLM framework" +description: "Create an OpenAI compatible endpoint using the vLLM framework" --- This tutorial creates an OpenAI-compatible endpoint that works with any open-source model. Use existing OpenAI code with Cerebrium serverless functions by changing just two lines of code. diff --git a/v4/examples/realtime-voice-agents.mdx b/v4/examples/realtime-voice-agents.mdx index 493de3fa..2e1a9380 100644 --- a/v4/examples/realtime-voice-agents.mdx +++ b/v4/examples/realtime-voice-agents.mdx @@ -27,7 +27,7 @@ Create a Cerebrium account by signing up [here](https://dashboard.cerebrium.ai/r See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium. - You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else + You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium else you must use their API endpoint below. diff --git a/v4/examples/sdxl.mdx b/v4/examples/sdxl.mdx index fcc0c82c..bc6920d8 100644 --- a/v4/examples/sdxl.mdx +++ b/v4/examples/sdxl.mdx @@ -153,7 +153,7 @@ curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/3-sdxl-refiner/predict' --data '{ "url": "https://huggingface.co/datasets/patrickvonplaten/images/resolve/main/aa_xl/000000009.png", "prompt": "a photo of an astronaut riding a horse on mars" -}'' +}' ``` The endpoint returns results in this format: diff --git a/v4/examples/transcribe-whisper.mdx b/v4/examples/transcribe-whisper.mdx index 6b0a230b..c1e20d60 100644 --- a/v4/examples/transcribe-whisper.mdx +++ b/v4/examples/transcribe-whisper.mdx @@ -149,7 +149,7 @@ After deployment, make this request: curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/1-whisper-transcription/predict' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ ---data '{"file_url": "https://your-public-url.com/test.mp3"}'' +--data '{"file_url": "https://your-public-url.com/test.mp3"}' ``` The response returns immediately with a 202 status code and a `run_id` — a unique identifier to correlate the result with the initial workload. diff --git a/v4/examples/twilio-voice-agent.mdx b/v4/examples/twilio-voice-agent.mdx index 3f12338b..b309ff3c 100644 --- a/v4/examples/twilio-voice-agent.mdx +++ b/v4/examples/twilio-voice-agent.mdx @@ -105,7 +105,7 @@ entrypoint = ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8765"] healthcheck_endpoint = "/health" ``` -You can read more about run custom web servers [here](https://docs.cerebrium.ai/container-images/custom-web-servers). +You can read more about running custom web servers [here](/container-images/custom-web-servers). ### Twilio setup