Skip to content

Add the Zoom AI Voice Agent service page #1584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cdliao wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add the Zoom AI Voice Agent service page #1584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e1be62f
Add the Zoom AI Voice Agent service page
cdliao Oct 23, 2026
2f46924
Document using your own OpenAI or Anthropic model with the Zoom AI Vo…
cdliao Oct 23, 2026
3e3288a
Note that rejected Zoom AI Voice Agent settings end the session
cdliao Oct 23, 2026
3327d58
Use ZoomAIRealtimeLLMService on the Zoom AI Voice Agent page
cdliao Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 242 additions & 0 deletions api-reference/server/services/s2s/zoom-ai.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
---
title: "Zoom AI Voice Agent"
sidebarTitle: "Zoom AI"
description: "Speech-to-speech conversation with ZoomAIRealtimeLLMService over the Zoom AI Services Voice Agent API, with function calling."
---

## Overview

`ZoomAIRealtimeLLMService` runs a speech-to-speech conversation with the [Voice Agent API](https://developers.zoom.us/docs/ai-services/voice-agent/) from Zoom AI Services, over a WebSocket [live session](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/). The Voice Agent detects turns, transcribes the caller, runs the model, and speaks the reply, so it replaces STT, the LLM, and TTS in a pipeline. It supports function calling, voice selection, custom vocabulary, and models from your own OpenAI or Anthropic account.

<CardGroup cols={2}>
<Card
title="Zoom AI Voice Agent API Reference"
icon="code"
href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.zoom_ai.realtime.llm.html"
>
Pipecat's API methods for Zoom AI Voice Agent integration
</Card>
<Card
title="Example Implementation"
icon="play"
href="https://github.com/pipecat-ai/pipecat/blob/main/examples/realtime/realtime-zoom-ai.py"
>
Voice agent with function calling
</Card>
<Card
title="Voice Agent Documentation"
icon="book"
href="https://developers.zoom.us/docs/ai-services/voice-agent/"
>
Official Zoom Voice Agent API documentation
</Card>
<Card title="Zoom AI Services" icon="microphone" href="https://www.zoom.ai/">
Get credentials and manage your Zoom AI Services plan
</Card>
</CardGroup>

## Installation

To use Zoom AI services, install the required dependency:

```bash
uv add "pipecat-ai[zoom-ai]"
```

## Prerequisites

### Zoom AI Account Setup

Before using the Zoom AI Voice Agent, you need:

1. **Zoom AI Services access**: Sign up through [Zoom AI Services](https://www.zoom.ai/)
2. **API key**: Create a Zoom AI Services API key. See [Get credentials](https://developers.zoom.us/docs/ai-services/auth/).

### Required Environment Variables

- `ZOOM_AI_API_KEY`: Your Zoom AI Services API key

## Configuration

### ZoomAIRealtimeLLMService

```python
from pipecat.services.zoom_ai.realtime.llm import ZoomAIRealtimeLLMService

Check failure on line 64 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.llm
```

<ParamField path="api_key" type="str" required>
Zoom AI Services API key. It is sent on every connection and reconnect, so it
must stay valid for as long as the service runs.
</ParamField>

<ParamField
path="url"
type="str"
default="wss://api.zoom.us/v2/ws/aiservices/voice-agent/live"
>
Voice Agent WebSocket URL.
</ParamField>

<ParamField
path="settings"
type="ZoomAIRealtimeLLMService.Settings"
default="None"
>
Runtime-configurable settings. See [Settings](#settings) below.
</ParamField>

<ParamField path="start_audio_paused" type="bool" default="False">
Whether to start with audio input paused.
</ParamField>

### Settings

Runtime-configurable settings passed via the `settings` constructor argument using `ZoomAIRealtimeLLMService.Settings(...)`. They map to the Voice Agent's [session settings](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#configure-the-session), and can be updated mid-conversation with `LLMUpdateSettingsFrame`. See [Service Settings](/pipecat/fundamentals/service-settings) for how settings work in Pipecat.

| Parameter | Type | Default | Description |
| -------------------- | ------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model` | `str` | `"gemma-4"` | Conversational model: a Zoom model, or a model from your own `provider`. Set when a session starts, so a change mid-conversation logs a warning and takes effect only after a reconnect. See [Think settings](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#think-settings). _(Inherited from base settings.)_ |
| `provider` | `Provider \| None` | `None` | Where the model runs. `None` uses Zoom's own models. Set it to use your own OpenAI or Anthropic account. Set when a session starts, like `model`. See [Provider](#provider). |
| `system_instruction` | `str \| None` | `None` | Instructions for the agent. `None` uses the Voice Agent's default instructions for spoken conversation. _(Inherited from base settings.)_ |
| `listen` | `Listen \| None` | `None` | How the agent listens: the caller's language and custom vocabulary. Set when a session starts, like `model`. `None` uses the Voice Agent's defaults. See [Listen](#listen). |
| `speak` | `Speak \| None` | `None` | How the agent speaks: its voice. `None` uses the Voice Agent's default voice. See [Speak](#speak). |
| `session_properties` | `SessionProperties` | `SessionProperties()` | OpenAI Realtime session properties. The service sends their `tools`, `tool_choice`, `max_output_tokens`, and `reasoning`. Set the voice with `speak`. |
| `extra` | `dict[str, Any]` | `{}` | Voice Agent session fields that have no setting, shaped like the `session` of [`session.update`](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#configure-the-session) and merged into it. |

`provider`, `listen`, and `speak` also accept plain dicts in the same shape, such as `{"voice": "ava"}`.

### Provider

```python
from pipecat.services.zoom_ai.realtime.events import Provider

Check failure on line 111 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.events
```

| Parameter | Type | Default | Description |
| ---------- | ---------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | `Literal["zoom", "openai", "anthropic"]` | `"zoom"` | `"zoom"` for Zoom's own models, or `"openai"` or `"anthropic"` for your own account with that provider. |
| `model` | `str \| None` | `None` | Model ID from the provider. `None` uses the service's `model` setting. |
| `headers` | `dict[str, str] \| None` | `None` | Your provider API key, as `{"X-Api-Key": "..."}`. Required for `"openai"` and `"anthropic"`. This is your OpenAI or Anthropic key, not your Zoom AI Services key. |
| `metadata` | `dict[str, Any] \| None` | `None` | The provider model's limits, such as `context_window` and `max_output_tokens`. |

See [Bring your own key](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#bring-your-own-key) for the providers and models the Voice Agent supports.

### Listen

```python
from pipecat.services.zoom_ai.realtime.events import Listen, Transcription, Vocabulary

Check failure on line 126 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.events
```

| Parameter | Type | Default | Description |
| -------------------------- | -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `transcription.language` | `str \| None` | `None` | Language of the caller. See [Listen settings](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#listen-settings) for supported languages. |
| `transcription.vocabulary` | `Vocabulary \| None` | `None` | Custom vocabulary: `Vocabulary(phrases=[...])`, names and terms to recognize, such as product names. |

The service sets the input audio format from the pipeline's `audio_in_sample_rate`.

### Speak

```python
from pipecat.services.zoom_ai.realtime.events import Speak

Check failure on line 139 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.events
```

| Parameter | Type | Default | Description |
| --------- | ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `voice` | `str \| None` | `None` | Voice ID, such as `"jenna"` or `"ava"`. `None` uses the Voice Agent's default voice. See [Available voices](https://developers.zoom.us/docs/ai-services/voice-agent/live-mode/#available-voices). |

## Usage

### Basic Setup

```python
import os

from pipecat.services.zoom_ai.realtime.llm import ZoomAIRealtimeLLMService

Check failure on line 153 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.llm

llm = ZoomAIRealtimeLLMService(
api_key=os.getenv("ZOOM_AI_API_KEY"),
settings=ZoomAIRealtimeLLMService.Settings(
system_instruction="You are a friendly assistant. Keep your replies to one or two sentences.",
),
)
```

Register tools on the `LLMContext`, as for other LLM services. See [Function Calling](/pipecat/learn/function-calling).

### Choosing a Voice and Vocabulary

This example sets the agent's voice and adds product names the Voice Agent should recognize:

```python
from pipecat.services.zoom_ai.realtime.events import Listen, Speak, Transcription, Vocabulary

Check failure on line 170 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.events

llm = ZoomAIRealtimeLLMService(
api_key=os.getenv("ZOOM_AI_API_KEY"),
settings=ZoomAIRealtimeLLMService.Settings(
system_instruction="You are the returns assistant for Zoom Cafe Shop.",
listen=Listen(
transcription=Transcription(
vocabulary=Vocabulary(phrases=["Zoom Cafe Shop", "house blend", "milk frother"])
)
),
speak=Speak(voice="ava"),
),
)
```

### Using Your Own OpenAI or Anthropic Model

To run the conversation on your own OpenAI or Anthropic account, set `provider` and name the provider's model in `model`:

```python
from pipecat.services.zoom_ai.realtime.events import Provider

Check failure on line 191 in api-reference/server/services/s2s/zoom-ai.mdx

View workflow job for this annotation

GitHub Actions / check-imports 

import path does not resolve: pipecat.services.zoom_ai.realtime.events

llm = ZoomAIRealtimeLLMService(
api_key=os.getenv("ZOOM_AI_API_KEY"),
settings=ZoomAIRealtimeLLMService.Settings(
model="gpt-4.1-mini",
provider=Provider(type="openai", headers={"X-Api-Key": os.getenv("OPENAI_API_KEY")}),
system_instruction="You are a friendly assistant.",
),
)
```

The Voice Agent still transcribes the caller and speaks the reply; only the model's turn uses your provider account.

### Updating Settings Mid-Conversation

Send an `LLMUpdateSettingsFrame` to change the instructions, tools, or voice during a conversation:

```python
from pipecat.frames.frames import LLMUpdateSettingsFrame

await worker.queue_frame(
LLMUpdateSettingsFrame(
delta=ZoomAIRealtimeLLMService.Settings(
system_instruction="You are the shipping assistant for Zoom Cafe Shop.",
speak=Speak(voice="mark"),
)
)
)
```

## Notes

- **No local VAD**: The Voice Agent detects turns and handles barge-in itself, so the pipeline needs no VAD analyzer. Turn detection isn't configurable.
- **Audio**: Input audio is sent at the pipeline's `audio_in_sample_rate`. The agent's audio is pushed as 24 kHz mono PCM. Audio that arrives while the session is connecting or reconnecting is held, up to the latest 3 seconds, and sent once the session is ready, so the caller can speak first.
- **Transcripts**: The caller's speech is pushed upstream as `TranscriptionFrame`s, and the agent's speech downstream as `TTSTextFrame`s.
- **Reconnects**: A session lasts up to 60 minutes. When a session ends unexpectedly, the service reconnects, up to 3 times in a row, and restores the conversation from the context. If reconnecting fails, it pushes a permanent error.
- **Errors**: If Zoom refuses the connection (for example, an invalid API key), the error includes Zoom's reason. If the Voice Agent rejects the session's settings (an unknown voice, for example), or the provider rejects your own provider key, the session ends with a permanent error. Other errors the Voice Agent reports without ending the session are pushed as non-fatal errors. Errors include the session's tracking ID (`x-zm-trackingid`), which Zoom support uses to find a session. See [Error Handling](/pipecat/fundamentals/error-handling).
- **System instruction precedence**: The `system_instruction` from service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.

## Event Handlers

| Event | Description |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `on_conversation_item_created` | Called when a new conversation item is created in the session |
| `on_conversation_item_updated` | Called when a conversation item is updated or completed. `item` is `None` when a user audio transcription completes. |

```python
@llm.event_handler("on_conversation_item_created")
async def on_item_created(service, item_id, item):
print(f"New conversation item: {item_id}")
```
1 change: 1 addition & 0 deletions api-reference/server/services/supported-services.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ Realtime LLMs (speech-to-speech) are multi-modal models that take in audio, vide
| [OpenAI Realtime](/api-reference/server/services/s2s/openai) | `uv add "pipecat-ai[openai]"` | Pipecat |
| [ThunderPhone](/api-reference/server/services/s2s/thunderphone) | `uv add pipecat-thunderphone` | Community |
| [Ultravox](/api-reference/server/services/s2s/ultravox) | `uv add "pipecat-ai[ultravox]"` | Pipecat |
| [Zoom AI Voice Agent](/api-reference/server/services/s2s/zoom-ai) | `uv add "pipecat-ai[zoom-ai]"` | Pipecat |

## Serializers

Expand Down
3 changes: 2 additions & 1 deletion docs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,8 @@
"api-reference/server/services/s2s/openai-live",
"api-reference/server/services/s2s/openai",
"api-reference/server/services/s2s/thunderphone",
"api-reference/server/services/s2s/ultravox"
"api-reference/server/services/s2s/ultravox",
"api-reference/server/services/s2s/zoom-ai"
]
},
{
Expand Down
Loading
Loading