Skip to content

Commit 55ef223

Browse files
xianshijing-lkclaudehiroshihorie
authored
Keep remote audio at media loudness when capture starts on iOS (#1280)
Reported by Proton (#ext-proton): on iOS, remote audio drops sharply as soon as local capture starts — unmuting, joining unmuted, or a prejoin mic preview. Users describe it as "join unmuted → quiet, join muted → fine". ## Root cause iOS keeps **separate media and call volume registers**, and which one feeds playback is selected by the audio session **mode**, not the category. Apple documents the mechanism on both `AVAudioSession.Mode.voiceChat` and `.videoChat`: > For apps that use one or more chat modes (voice, video, or game), but don't use > Audio Unit Voice I/O or AVAudioEngine with `setVoiceProcessingEnabled(_:)`, the > system reduces the processing it applies to audio signals. Specifically, it > doesn't apply voice-specific processing, like echo cancellation and automatic > gain correction, and disables dynamic processing on input and output, **which > results in a lower playback level**. In automatic mode the SDK resolved the session from engine state — playout-only to `playback`/`spokenAudio`, recording to `playAndRecord`/`videoChat` — so starting capture moved remote playback onto the call register. When the user's call volume sits below their media volume, remote audio plays at a fraction of the expected level. Measured on an iPhone 14 / iOS 27.0.1, with both volume registers seeded programmatically so no hardware button is involved: | | recording | category / mode | `outputVolume` | register | |---|---|---|---|---| | joined muted | no | `Playback`/`SpokenAudio` | 1.00 | media | | unmuted | yes | `PlayAndRecord`/`VideoChat` | 0.25 | call | | manual `playAndRecord` + `default`, playout only | no | `PlayAndRecord`/`Default` | 1.00 | media | | same, capturing | yes | `PlayAndRecord`/**`VoiceChat`** ← iOS rewrote it | 0.25 | call | The third and fourth rows isolate it: same category, `default` mode, media register — and iOS rewrites `default` back to `voiceChat` the moment Voice Processing I/O is instantiated. So a session mode alone cannot avoid the call-tuned gain; VPIO has to actually be out of the picture. ## Fix Ports [client-sdk-swift#1072](livekit/client-sdk-swift#1072) (Swift SDK 2.16.0), which solved the same problem. - A capture that resolves to WebRTC **software** voice processing now gets `playAndRecord` with the `default` mode and keeps media loudness. With Apple voice processing the chat mode stays, because VPIO supplies the compensating loudness stage. - **`AudioManager.setPlatformVoiceProcessingAllowed(bool)`** (new) guarantees VPIO is never used — which is what lets `default` stick. - `playAndRecord` is kept across mute toggles until the engine stops, instead of flipping back to `playback`. That flip moved playback between the two registers and tore down echo cancellation mid-call. - `bypassVoiceProcessing: true` now tells the audio device module what it already tells the `getUserMedia` constraints. It previously left Apple voice processing requested, leaving the session on a chat mode whose reduced playback level had nothing left to compensate it — measured at close to inaudible. - The automatic speaker policy now sets `defaultToSpeaker`, which `default` mode needs to stay off the receiver (the chat modes apply it automatically). Verified on device — capturing with the policy opted out: | step | recording | category / mode | `outputVolume` | aec | |---|---|---|---|---| | capturing, VP disallowed | yes | `PlayAndRecord`/`Default` | **1.00** | `software` active | | muted | yes | `PlayAndRecord`/`Default` | 1.00 | `software` active | | unmuted | yes | `PlayAndRecord`/`Default` | 1.00 | `software` active | Echo cancellation is genuinely running in software — not "AEC off". ## API Additive and opt-in; `dart-apitool diff` against `main` reports **no breaking changes**. The default path is deliberately unchanged: with Apple voice processing allowed, a capturing session still follows the call register. Opting out trades Apple's per-device-tuned echo cancellation for WebRTC's, so it should be an app's explicit choice. ```dart await AudioManager.instance.setPlatformVoiceProcessingAllowed(false); final allowed = await AudioManager.instance.isPlatformVoiceProcessingAllowed(); ``` ## Tests These decisions lived inside `LKAudioEngineObserver`, which imports Flutter and WebRTC, and this repo has no native test target — so the riskiest part of the fix had no automated coverage. They are now extracted into `LKAudioSessionPolicyResolver`, carrying no Flutter, WebRTC or AVFoundation types (`AVAudioSession` does not exist on macOS), so the state matrix runs as a plain host Swift package. The observer keeps only the mapping onto `RTCAudioSessionConfiguration`, so the tested logic is the shipped logic. - `swift test --package-path test/native` — 17 tests. Manual vs automatic management, playout-only, capture with and without Apple voice processing, muting mid-call, engine restart, coupled AEC/NS resolution, plus the reported sequence replayed end to end. - `flutter test` — 515 tests (+4), covering the capture-options map that decides this natively. - New `native-test-check` CI job on `macos-latest`: no simulator, device or pod install, so it gates every PR. ## Known limitations - **Per-capture `AudioProcessingMode.software` is still ignored by the audio device module.** Verified on device: `isPlatformActive: true`, `isSoftwareActive: false`. Only the policy above reliably keeps VPIO away. The authoritative fix belongs in the webrtc-sdk fork — the ADM should honour the request or reject it with a clear error. Documented on the API for now. A visible consequence: when the ADM ignores a software request, the session policy and the actual processing path disagree (`default` mode with VPIO active). - **Routes are not covered.** Measurements are one device, built-in speaker and receiver. Bluetooth HFP, wired headsets and mid-call route changes still need manual verification — note that `videoChat` restricts the permitted route set, so the `default`-mode path allows routes the chat modes do not. - The sticky `playAndRecord` is the one part that changes behaviour for every iOS app with no opt-in: a muted stretch no longer returns to `playback`, so it loses `mixWithOthers` and stays on Bluetooth HFP rather than A2DP. It is separable from the rest of the fix if we would rather not take that trade now. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>
1 parent 4dae650 commit 55ef223

19 files changed

Lines changed: 1343 additions & 12 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
minor type="fixed" "iOS: remote audio no longer has to drop to the device's call volume as soon as local capture starts. iOS applies a reduced, call-tuned output gain while the audio session runs a chat mode (`videoChat`/`voiceChat`), which Apple's Voice Processing I/O compensates for; without VPIO there is nothing to compensate, so a capture that resolves to WebRTC software voice processing now gets `playAndRecord` with the `default` mode and keeps media loudness. The new `AudioManager.setPlatformVoiceProcessingAllowed` guarantees Voice Processing I/O is never used, which is what lets the `default` mode stick — iOS rewrites the mode back to `voiceChat` whenever VPIO is instantiated, so a session mode alone cannot avoid the call-tuned gain. In automatic mode `playAndRecord` is additionally kept across mute toggles until the engine stops, instead of flipping back to `playback` on every mute: that flip moved playback between iOS's media and call volume registers (audible as a loud-then-quiet jump) and tore down echo cancellation mid-call. `LiveKitClient.initialize(bypassVoiceProcessing: true)` now also tells the audio device module to drop processing, matching what it already does to the `getUserMedia` constraints; previously it left Apple voice processing requested, so the session kept a chat mode whose call-tuned gain had nothing left to compensate it and remote audio became close to inaudible. The automatic speaker policy now also sets `defaultToSpeaker`, which the `default` mode needs to stay on the speaker"

‎.github/workflows/build.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,19 @@ jobs:
8484
- name: Dart Test Check
8585
run: flutter test
8686

87+
native-test-check:
88+
name: Native Test
89+
# The audio-session policy decisions are made in shared_swift and carry no
90+
# Flutter, WebRTC or AVFoundation dependency, so they run as a plain host
91+
# Swift package -- no simulator, device or pod install.
92+
runs-on: macos-latest
93+
94+
steps:
95+
- name: Clone repository
96+
uses: actions/checkout@v6
97+
- name: Native Test Check
98+
run: swift test --package-path test/native
99+
87100
build-for-android:
88101
name: Android
89102
runs-on: ubuntu-latest

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,3 +89,6 @@ lib/generated_plugin_registrant.dart
8989

9090
# Test files - ignore any binary files in testfiles directory
9191
testfiles/*.bin
92+
93+
# SwiftPM build output for the native policy tests (test/native)
94+
.build/
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../shared_swift/AudioSessionPolicyResolver.swift
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../../../shared_swift/AudioSessionPolicyResolver.swift

‎lib/livekit_client.dart‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ export 'src/publication/track_publication.dart';
5959
export 'src/support/platform.dart';
6060
export 'src/audio/audio_processing_state.dart';
6161
export 'src/track/audio_visualizer.dart';
62-
export 'src/track/local/audio.dart';
62+
export 'src/track/local/audio.dart' hide resolveCaptureProcessingOptions;
6363
export 'src/track/local/local.dart';
6464
export 'src/track/local/video.dart';
6565
export 'src/track/options.dart';

‎lib/src/audio/audio_manager.dart‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,75 @@ class AudioManager {
459459
}
460460
}
461461

462+
/// Whether Apple's platform voice processing is allowed. Always true off
463+
/// iOS/macOS, where the policy does not apply.
464+
///
465+
/// Experimental: this API may change in a future release.
466+
@experimental
467+
Future<bool> isPlatformVoiceProcessingAllowed() async {
468+
if (!lkPlatformIsApple()) return true;
469+
return Native.isPlatformVoiceProcessingAllowed();
470+
}
471+
472+
/// Sets whether Apple's platform voice processing (Voice Processing I/O) may
473+
/// be used. Defaults to true. A no-op off iOS/macOS.
474+
///
475+
/// When set to false, voice processing resolves to WebRTC's software
476+
/// implementation: [AudioProcessingMode.automatic] falls back to software and
477+
/// [AudioProcessingMode.platform] is rejected. Any VPIO path currently
478+
/// requested is torn down.
479+
///
480+
/// Use this when the app must guarantee Voice Processing I/O is never used —
481+
/// most often to keep remote playback at media loudness on iOS. Apple
482+
/// documents that a session using a chat mode (`voiceChat`/`videoChat`/
483+
/// `gameChat`) without Voice Processing I/O gets reduced processing and "a
484+
/// lower playback level", so LiveKit only keeps a chat mode while VPIO is
485+
/// supplying the compensating loudness stage. Disallowing VPIO moves capture
486+
/// to the `default` mode, which stays on the device's media volume instead of
487+
/// its separate call volume.
488+
///
489+
/// This is a process-wide policy, not a per-capture preference: on iOS the
490+
/// audio device module and the audio session are shared by every track, so
491+
/// two local tracks cannot resolve to different voice processing
492+
/// implementations. [AudioCaptureOptions]' `*Mode` fields remain the
493+
/// per-capture expression of the same intent, but today the audio device
494+
/// module resolves a capture-time [AudioProcessingMode.software] request back
495+
/// to the platform path, so only this policy reliably keeps Voice Processing
496+
/// I/O out of the way. Verify either with [getAudioProcessingState], which
497+
/// reports requested versus effective implementation.
498+
///
499+
/// Set this before capture starts — ideally once at startup, before
500+
/// connecting. Changing it mid-capture tears down the live VPIO path and
501+
/// reconfigures the session underneath an active call.
502+
///
503+
/// Like [setMicrophoneMuteMode] this reaches the audio device module, which
504+
/// exists once `LiveKitClient.initialize()` has run (connecting a room runs
505+
/// it too). Calling either of these before that throws
506+
/// `rejectedPlatformUnavailable`, so an app setting this at startup should
507+
/// await `LiveKitClient.initialize()` first.
508+
///
509+
/// The trade-off is deliberate: WebRTC's software echo cancellation replaces
510+
/// Apple's, which is tuned per device and generally performs better on the
511+
/// built-in speaker and microphone. Prefer leaving platform voice processing
512+
/// allowed unless media playback loudness during capture matters more.
513+
///
514+
/// Throws if the native side rejects the change, so callers never assume
515+
/// software processing is in effect when it is not. Like
516+
/// [setMicrophoneMuteMode] this can rebuild the audio engine, so it throws a
517+
/// [TrackCreateException] when microphone permission is missing and an
518+
/// [AudioSessionException] when the audio session does not permit recording.
519+
///
520+
/// Experimental: this API may change in a future release.
521+
@experimental
522+
Future<void> setPlatformVoiceProcessingAllowed(bool allowed) async {
523+
if (!lkPlatformIsApple()) return;
524+
try {
525+
await Native.setPlatformVoiceProcessingAllowed(allowed);
526+
} on PlatformException catch (error) {
527+
throw audioEngineExceptionFrom(error) ?? error;
528+
}
529+
}
530+
462531
/// Diagnostic snapshot of the resolved audio processing state.
463532
///
464533
/// The audio processing module is owned by the native peer connection factory

‎lib/src/audio/audio_session_policy.dart‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ class ResolvedAudioSessionPolicy {
3939
AppleAudioCategoryOption.allowBluetooth,
4040
AppleAudioCategoryOption.allowBluetoothA2DP,
4141
AppleAudioCategoryOption.allowAirPlay,
42+
// The chat modes imply a speaker route, but native replaces the mode
43+
// with `default` when the capture does not use Apple voice
44+
// processing, and `default` routes to the receiver without this. A
45+
// connected headset still wins over it.
46+
if (preferSpeakerOutput) AppleAudioCategoryOption.defaultToSpeaker,
4247
},
4348
appleAudioMode: preferSpeakerOutput ? AppleAudioMode.videoChat : AppleAudioMode.voiceChat,
4449
);

‎lib/src/support/native.dart‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,28 @@ class Native {
174174
return null;
175175
}
176176

177+
/// Reads the app-level policy for Apple's platform voice processing.
178+
@internal
179+
static Future<bool> isPlatformVoiceProcessingAllowed() async {
180+
final response = await channel.invokeMethod<bool>(
181+
'isPlatformVoiceProcessingAllowed',
182+
<String, dynamic>{},
183+
);
184+
return response ?? true;
185+
}
186+
187+
/// Sets the app-level policy for Apple's platform voice processing.
188+
///
189+
/// Errors propagate: a caller that disallowed the platform path must not be
190+
/// left believing software processing is in effect when it is not.
191+
@internal
192+
static Future<void> setPlatformVoiceProcessingAllowed(bool allowed) async {
193+
await channel.invokeMethod<void>(
194+
'setPlatformVoiceProcessingAllowed',
195+
<String, dynamic>{'allowed': allowed},
196+
);
197+
}
198+
177199
/// Configure and activate LiveKit's Android audio session (mode/focus/routing).
178200
@internal
179201
static Future<void> configureAndroidAudioSession(Map<String, dynamic> configuration) async {

‎lib/src/track/local/audio.dart‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,19 @@ import '../audio_management.dart';
3535
import '../options.dart' as track_options;
3636
import 'local.dart';
3737

38+
/// The processing options the audio device module is started with for [options].
39+
///
40+
/// `bypassVoiceProcessing` already strips every processing constraint in
41+
/// [AudioCaptureOptions.toMediaConstraintsMap], so the audio device module has
42+
/// to hear the same thing. Otherwise it keeps Apple voice processing requested
43+
/// and the session stays on a chat mode, where Apple documents that an app not
44+
/// using Voice Processing I/O gets reduced processing and "a lower playback
45+
/// level" — leaving remote audio close to inaudible.
46+
@internal
47+
track_options.AudioProcessingOptions resolveCaptureProcessingOptions(
48+
track_options.AudioCaptureOptions options,
49+
) => Native.bypassVoiceProcessing ? const track_options.AudioProcessingOptions.noProcessing() : options.processing;
50+
3851
class LocalAudioTrack extends LocalTrack with AudioTrack, LocalAudioManagementMixin {
3952
// Options used for this track
4053
@override
@@ -96,7 +109,7 @@ class LocalAudioTrack extends LocalTrack with AudioTrack, LocalAudioManagementMi
96109
try {
97110
// Match Swift: start the ADM before publishing so capture-time audio
98111
// processing options are applied before WebRTC opens the microphone.
99-
await Native.startLocalRecording(currentOptions.processing.toMap());
112+
await Native.startLocalRecording(resolveCaptureProcessingOptions(currentOptions).toMap());
100113
} on PlatformException catch (error) {
101114
// Missing microphone permission or an audio session that does not
102115
// permit recording are not audio processing failures, so they surface

0 commit comments

Comments
 (0)