Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions client/rtvi-standard.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -391,27 +391,35 @@ What the user said or typed. The observer sends one for each transcription (befo

#### bot-output 🤖

The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes a `text_type` field to indicate what the text represents (e.g. "sentence", "word", "code", "url") and fields describing whether and how far the bot has spoken it.
The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes an `output_type` field to indicate what the text represents (e.g. "sentence", "word", "code", "url") and fields describing whether and how far the bot has spoken it.

The bot's output is split into segments, such as sentences, each with its own `segment_id`. For clients of protocol 2.0.0 or later, a spoken segment is reported as it is spoken:

- If the TTS service reports word timestamps, the server sends the segment once with `spoken_status: "new"` before it is spoken, then again as each word is spoken, with `spoken_status: "in-progress"` and the last with `"completed"`. Every message carries the segment's full `text` and the same `segment_id`.
- Otherwise, the server sends the segment once, with `spoken_status: "completed"`, after it is spoken.

Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_status: "new"`. Clients of protocol 2.0.0 or later don't receive separate `bot-output` messages for individual words (`text_type` `"word"`); word-level progress is in `spoken_progress` instead. Clients on 1.x receive each spoken word as its own message and should read `spoken`.
Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_status: "new"`. Clients of protocol 2.0.0 or later don't receive separate `bot-output` messages for individual words (`output_type` `"word"`); word-level progress is in `spoken_progress` instead. Clients on 1.x receive each spoken word as its own message and should read `spoken`.

- **type**: `'bot-output'`
- **data**:
- **text**: `string`

The output text from the bot.

- **text_type**: `string`
- **output_type**: `string`

Indicates what kind of text this is (e.g., "sentence", "word", "code", "url"). "sentence", "word", and "backchannel" are reserved text types defined by the RTVI standard. Other text types may be defined by custom text aggregators used by the server.
Indicates what kind of text this is (e.g., "sentence", "word", "code", "url"). "sentence", "word", and "backchannel" are reserved output types defined by the RTVI standard. Other output types may be defined by custom text aggregators used by the server.

A short acknowledgment the bot says, such as "Mm-hmm.", has type `backchannel`. It is only sent to clients of protocol 2.2.0 or later.

- **text_type**: `string` (deprecated)

Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

- **aggregated_by**: `string` (deprecated)

Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

- **segment_id**: `number` (optional)

Identifies the segment this text belongs to, so a client can match a segment's `"new"`, `"in-progress"`, and `"completed"` messages.
Expand Down Expand Up @@ -439,10 +447,6 @@ Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_

Indicates if this text was spoken by the bot. Clients of protocol 2.0.0 or later should read `will_be_spoken` and `spoken_status` instead.

- **aggregated_by**: `string`

Deprecated. Use `text_type` instead. The server sends both fields with the same value.

#### bot-transcription 🤖

<Warning>
Expand Down
20 changes: 12 additions & 8 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22835,27 +22835,35 @@ What the user said or typed. The observer sends one for each transcription (befo

#### bot-output 🤖

The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes a `text_type` field to indicate what the text represents (e.g. "sentence", "word", "code", "url") and fields describing whether and how far the bot has spoken it.
The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes an `output_type` field to indicate what the text represents (e.g. "sentence", "word", "code", "url") and fields describing whether and how far the bot has spoken it.

The bot's output is split into segments, such as sentences, each with its own `segment_id`. For clients of protocol 2.0.0 or later, a spoken segment is reported as it is spoken:

- If the TTS service reports word timestamps, the server sends the segment once with `spoken_status: "new"` before it is spoken, then again as each word is spoken, with `spoken_status: "in-progress"` and the last with `"completed"`. Every message carries the segment's full `text` and the same `segment_id`.
- Otherwise, the server sends the segment once, with `spoken_status: "completed"`, after it is spoken.

Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_status: "new"`. Clients of protocol 2.0.0 or later don't receive separate `bot-output` messages for individual words (`text_type` `"word"`); word-level progress is in `spoken_progress` instead. Clients on 1.x receive each spoken word as its own message and should read `spoken`.
Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_status: "new"`. Clients of protocol 2.0.0 or later don't receive separate `bot-output` messages for individual words (`output_type` `"word"`); word-level progress is in `spoken_progress` instead. Clients on 1.x receive each spoken word as its own message and should read `spoken`.

- **type**: `'bot-output'`
- **data**:
- **text**: `string`

The output text from the bot.

- **text_type**: `string`
- **output_type**: `string`

Indicates what kind of text this is (e.g., "sentence", "word", "code", "url"). "sentence", "word", and "backchannel" are reserved text types defined by the RTVI standard. Other text types may be defined by custom text aggregators used by the server.
Indicates what kind of text this is (e.g., "sentence", "word", "code", "url"). "sentence", "word", and "backchannel" are reserved output types defined by the RTVI standard. Other output types may be defined by custom text aggregators used by the server.

A short acknowledgment the bot says, such as "Mm-hmm.", has type `backchannel`. It is only sent to clients of protocol 2.2.0 or later.

- **text_type**: `string` (deprecated)

Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

- **aggregated_by**: `string` (deprecated)

Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

- **segment_id**: `number` (optional)

Identifies the segment this text belongs to, so a client can match a segment's `"new"`, `"in-progress"`, and `"completed"` messages.
Expand Down Expand Up @@ -22883,10 +22891,6 @@ Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_

Indicates if this text was spoken by the bot. Clients of protocol 2.0.0 or later should read `will_be_spoken` and `spoken_status` instead.

- **aggregated_by**: `string`

Deprecated. Use `text_type` instead. The server sends both fields with the same value.

#### bot-transcription 🤖

<Warning>
Expand Down
Loading