Skip to content

Commit f38055a

Browse files
1egomanhiroshihorieclaude
authored
Data streams v2 (#1166)
Initial port of data streams v2 to the flutter sdk. This is built on top of #1160, and that PR must be merged first. ## Summary <details><summary>Background context on data streams v2</summary> <p> Previously, the flutter sdk had its own data streams implementation. With [data streams v2](livekit/client-sdk-js#1985), the rust implementation will both be quite a bit more stable and gain some new features that make it significantly more performant (single packet data streams and DEFLATE compression when it makes the payload smaller). This roughly _doubles_ data stream throughput in local testing. </p> </details> In the corresponding [swift](livekit/client-sdk-swift#1075) and [android](livekit/client-sdk-android#997) pull requests, I was able to fully cut over to data streams v2. However, the flutter sdk also has to support web, and today uniffi-dart does not support WASM - see [here](https://redirect.github.com/Uniffi-Dart/uniffi-dart/issues/106) for an early issue on the bindgen side tracking this work. So in lieu of fully cutting over: - All platforms except for web use the rust provided data streams v2 implementation - Web keeps the existing pure dart data streams v1 implementation. Note that it could be possible to make flutter web use the web sdk's data stream v2 implementation, but it looks like it would introduce a lot of infrastructure complexity. Given this and that it seems like flutter web support is somewhat experimental anyway, I'm tempted to keep it on the existing implementation. ### New behaviors worth being aware of All of these are either data streams v2 related changes, or bug fixes. #### 1. `compress` option When sending a data stream, there is a new `compress` option. Just like how this works on web / rust, this defaults to `true`. If set to `false`, then compression will be disabled (useful if you know the data you are sending isn't compressible / you are doing your own compression, which is not uncommon in robotics use cases). The vast majority of users should leave this set to `true`. #### 2. Max data stream size In a [rust data streams v2 pull request review comment](livekit/rust-sdks#1192 (comment)), we decided that for security reasons it made sense to introduce a maximum data stream size as a DOS protection. This limit is by default `5gb` - any data stream that is larger will now read up until that point, and if the stream keeps going, a "payload too large" error will be raised on the stream and exposed to a user on the subsequent `.read()` call. If a user is sending a large file, they can override this by setting a new `maxPayloadByteLength` option on the `room.connect` call: ```kotlin await room.connect( wsUrl, token, connectOptions: const ConnectOptions( autoSubscribe: true, dataStream: const DataStreamOptions(maxPayloadByteLength = 1000), ), ); ``` #### 3. Throwing new data stream errors types The old flutter specific data streams implementation wasn't quite as strict and didn't surface as many error cases to the caller which were encountered while reading the stream as the rust implementation now does. This needs some testing in some of these edge cases to make sure I didn't inadvertently make this backwards incompatible in a non-aceptable way. ----------------- > [!WARNING] > This pull request was LLM generated and has only been reviewed by a human who isn't a domain expert in android development. I have tested this and confirms it works in the happy path, but no other validation has been done. > > A more thorough review of this needs to occur before it could be merged. ## Todo - [ ] Get #1160 merged and released. - [ ] Drop c14d0e7 once #1169 is merged, this commit duplicates that fix in here - [ ] Test this thoroughly. I've run a happy path test and that's it. I'm holding off on doing more until #1160 is in a more stable place. - [x] Add changeset --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 81a79ad commit f38055a

30 files changed

Lines changed: 2697 additions & 538 deletions

‎.changes/data-stream-options‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
minor type="added" "ConnectOptions.dataStream with maxPayloadByteLength, bounding the payload a single incoming data stream may deliver"

‎.changes/data-streams-v2‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
minor type="changed" "Data streams are now backed by the Rust core (livekit-ffi) on native platforms, adding data streams v2: single-packet inline sends, deflate-raw compression and MTU-bounded headers. Adds LocalParticipant.sendBytes, a compress option, Participant.capabilities and ClientProtocolVersion.v2. Web keeps the existing Dart implementation and interoperates as a pre-v2 peer."

‎.changes/stream-packet-after-close‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
patch type="fixed" "A data stream packet that arrives after the room closed no longer opens a stream that nothing finishes or fails"

‎.changes/stream-writer-close‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
patch type="fixed" "Closing a data stream writer more than once ends the stream once instead of sending another trailer, and a write after close fails with DataStreamErrorReason.AlreadyClosed instead of sending a chunk after the stream ended"

‎AGENTS.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,16 @@ Web/native divergence is handled with conditional imports (e.g. `track/processor
3636

3737
`lib/src/ffi/` wraps `livekit_ffi`, a Dart package generated from the `livekit-ffi` crate in the sibling `rust-sdks` repo. It reaches Rust through Dart's Native Assets: the package's `hook/build.dart` bundles a `cdylib` into the host app and the generated bindings call into it with `@Native`. This is why the SDK requires Flutter >= 3.38 / Dart >= 3.10.
3838

39-
There is no dynamic library to load on the web, so `ffi.dart` splits native/web the same way the rest of the SDK does. **`ffi_io.dart` is the only file allowed to import `package:livekit_ffi/...`** — importing it from anywhere reachable on web pulls `dart:ffi` into a web compile and breaks `flutter build web`/`--wasm`. Guard calls with `LiveKitFFI.isAvailable`.
39+
There is no dynamic library to load on the web, so `ffi.dart` splits native/web the same way the rest of the SDK does. **Only `ffi_io.dart` and files under `lib/src/data_stream/` whose names end in `_native.dart` (plus `ffi_bridged.dart`) may import `package:livekit_ffi/...`** — importing it from anywhere reachable on web pulls `dart:ffi` into a web compile and breaks `flutter build web`/`--wasm`. No generated `livekit_ffi` type may appear in a public API signature; convert at the boundary (`data_stream/ffi_bridged.dart`). Guard calls with `LiveKitFFI.isAvailable`.
40+
41+
### Data streams
42+
43+
`lib/src/data_stream/` has two implementations behind one interface (`data_streams.dart`, conditional import): `data_streams_native.dart` delegates to the Rust core, which implements **data streams v2** (inline single-packet sends, deflate-raw compression, UTF-8-aware chunking, MTU-bounded headers); `data_streams_web.dart` is the original Dart v1 code, kept because the cdylib can't run in a browser. Web advertises `ClientProtocolVersion.v1` and no capabilities, so v2 senders fall back to uncompressed multi-packet for it.
44+
45+
Two things to know when touching the native path:
46+
47+
- **The core's push delegates cannot be used from Dart.** uniffi compiles a callback interface to `Pointer.fromFunction`, which is only valid on the isolate's thread, and the core invokes those delegates from its tokio runtime — the VM aborts with `Cannot invoke native callback outside an isolate`, which is not catchable. The managers are therefore built through the crate's `polled*` adapters (`livekit-ffi/src/core_modules/data_stream/polled.rs`), which implement the delegates *in Rust*, buffer into a channel, and expose an `async fn next_*` we await. `RemoteParticipantRegistryDelegate` is the one callback we implement directly, and it is safe: it is only called synchronously inside a `send*` future, which uniffi polls from the calling (Dart) thread.
48+
- **Whoever awaits a `livekit_ffi` object is the only thing that may dispose it.** Freeing the Rust handle while a `next()`/`nextPackets()` is in flight is a use-after-free that surfaces as a SIGBUS with no Dart stack. Hence readers are disposed by their pump rather than from a subscription's `onCancel`, and `dispose()` calls `close()` on the queues to wake their pumps instead of releasing them directly.
4049

4150
### Local development loop
4251

‎lib/livekit_client.dart‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ export 'src/connection_check/events.dart';
2222
export 'src/constants.dart';
2323
export 'src/core/room.dart';
2424
export 'src/core/room_preconnect.dart';
25+
export 'src/data_stream/errors.dart';
2526
export 'src/data_stream/stream_reader.dart';
2627
export 'src/data_stream/stream_writer.dart';
2728
export 'src/e2ee/e2ee_manager.dart';
@@ -69,6 +70,7 @@ export 'src/track/remote/remote.dart';
6970
export 'src/track/remote/video.dart';
7071
export 'src/track/track.dart';
7172
export 'src/json/agent_attributes.dart';
73+
export 'src/types/client_capability.dart';
7274
export 'src/types/data_stream.dart';
7375
export 'src/types/audio_encoding.dart';
7476
export 'src/types/other.dart';

‎lib/src/core/engine.dart‎

Lines changed: 6 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1166,29 +1166,13 @@ class Engine extends Disposable with EventsEmittable<EngineEvent> {
11661166
identity: dp.participantIdentity,
11671167
),
11681168
);
1169-
} else if (dp.whichValue() == lk_models.DataPacket_Value.streamHeader) {
1170-
// Data Stream Header
1169+
} else if (dp.whichValue() == lk_models.DataPacket_Value.streamHeader ||
1170+
dp.whichValue() == lk_models.DataPacket_Value.streamChunk ||
1171+
dp.whichValue() == lk_models.DataPacket_Value.streamTrailer) {
1172+
// Data stream header / chunk / trailer, forwarded whole — see EngineDataStreamPacketEvent.
11711173
events.emit(
1172-
EngineDataStreamHeaderEvent(
1173-
header: dp.streamHeader,
1174-
identity: dp.participantIdentity,
1175-
encryptionType: encryptionType,
1176-
),
1177-
);
1178-
} else if (dp.whichValue() == lk_models.DataPacket_Value.streamChunk) {
1179-
// Data Stream Chunk
1180-
events.emit(
1181-
EngineDataStreamChunkEvent(
1182-
chunk: dp.streamChunk,
1183-
identity: dp.participantIdentity,
1184-
encryptionType: encryptionType,
1185-
),
1186-
);
1187-
} else if (dp.whichValue() == lk_models.DataPacket_Value.streamTrailer) {
1188-
// Data Stream trailer
1189-
events.emit(
1190-
EngineDataStreamTrailerEvent(
1191-
trailer: dp.streamTrailer,
1174+
EngineDataStreamPacketEvent(
1175+
packet: dp,
11921176
identity: dp.participantIdentity,
11931177
encryptionType: encryptionType,
11941178
),

0 commit comments

Comments
 (0)