Repository navigation
Commit 55ef223
Keep remote audio at media loudness when capture starts on iOS (#1280)
Reported by Proton (#ext-proton): on iOS, remote audio drops sharply as
soon as
local capture starts — unmuting, joining unmuted, or a prejoin mic
preview.
Users describe it as "join unmuted → quiet, join muted → fine".
## Root cause
iOS keeps **separate media and call volume registers**, and which one
feeds
playback is selected by the audio session **mode**, not the category.
Apple
documents the mechanism on both `AVAudioSession.Mode.voiceChat` and
`.videoChat`:
> For apps that use one or more chat modes (voice, video, or game), but
don't use
> Audio Unit Voice I/O or AVAudioEngine with
`setVoiceProcessingEnabled(_:)`, the
> system reduces the processing it applies to audio signals.
Specifically, it
> doesn't apply voice-specific processing, like echo cancellation and
automatic
> gain correction, and disables dynamic processing on input and output,
**which
> results in a lower playback level**.
In automatic mode the SDK resolved the session from engine state —
playout-only
to `playback`/`spokenAudio`, recording to `playAndRecord`/`videoChat` —
so
starting capture moved remote playback onto the call register. When the
user's
call volume sits below their media volume, remote audio plays at a
fraction of
the expected level.
Measured on an iPhone 14 / iOS 27.0.1, with both volume registers seeded
programmatically so no hardware button is involved:
| | recording | category / mode | `outputVolume` | register |
|---|---|---|---|---|
| joined muted | no | `Playback`/`SpokenAudio` | 1.00 | media |
| unmuted | yes | `PlayAndRecord`/`VideoChat` | 0.25 | call |
| manual `playAndRecord` + `default`, playout only | no |
`PlayAndRecord`/`Default` | 1.00 | media |
| same, capturing | yes | `PlayAndRecord`/**`VoiceChat`** ← iOS rewrote
it | 0.25 | call |
The third and fourth rows isolate it: same category, `default` mode,
media
register — and iOS rewrites `default` back to `voiceChat` the moment
Voice
Processing I/O is instantiated. So a session mode alone cannot avoid the
call-tuned gain; VPIO has to actually be out of the picture.
## Fix
Ports
[client-sdk-swift#1072](livekit/client-sdk-swift#1072)
(Swift SDK 2.16.0), which solved the same problem.
- A capture that resolves to WebRTC **software** voice processing now
gets
`playAndRecord` with the `default` mode and keeps media loudness. With
Apple
voice processing the chat mode stays, because VPIO supplies the
compensating
loudness stage.
- **`AudioManager.setPlatformVoiceProcessingAllowed(bool)`** (new)
guarantees
VPIO is never used — which is what lets `default` stick.
- `playAndRecord` is kept across mute toggles until the engine stops,
instead of
flipping back to `playback`. That flip moved playback between the two
registers
and tore down echo cancellation mid-call.
- `bypassVoiceProcessing: true` now tells the audio device module what
it already
tells the `getUserMedia` constraints. It previously left Apple voice
processing
requested, leaving the session on a chat mode whose reduced playback
level had
nothing left to compensate it — measured at close to inaudible.
- The automatic speaker policy now sets `defaultToSpeaker`, which
`default` mode
needs to stay off the receiver (the chat modes apply it automatically).
Verified on device — capturing with the policy opted out:
| step | recording | category / mode | `outputVolume` | aec |
|---|---|---|---|---|
| capturing, VP disallowed | yes | `PlayAndRecord`/`Default` | **1.00**
| `software` active |
| muted | yes | `PlayAndRecord`/`Default` | 1.00 | `software` active |
| unmuted | yes | `PlayAndRecord`/`Default` | 1.00 | `software` active |
Echo cancellation is genuinely running in software — not "AEC off".
## API
Additive and opt-in; `dart-apitool diff` against `main` reports **no
breaking
changes**. The default path is deliberately unchanged: with Apple voice
processing allowed, a capturing session still follows the call register.
Opting
out trades Apple's per-device-tuned echo cancellation for WebRTC's, so
it should
be an app's explicit choice.
```dart
await AudioManager.instance.setPlatformVoiceProcessingAllowed(false);
final allowed = await AudioManager.instance.isPlatformVoiceProcessingAllowed();
```
## Tests
These decisions lived inside `LKAudioEngineObserver`, which imports
Flutter and
WebRTC, and this repo has no native test target — so the riskiest part
of the fix
had no automated coverage. They are now extracted into
`LKAudioSessionPolicyResolver`, carrying no Flutter, WebRTC or
AVFoundation types
(`AVAudioSession` does not exist on macOS), so the state matrix runs as
a plain
host Swift package. The observer keeps only the mapping onto
`RTCAudioSessionConfiguration`, so the tested logic is the shipped
logic.
- `swift test --package-path test/native` — 17 tests. Manual vs
automatic
management, playout-only, capture with and without Apple voice
processing,
muting mid-call, engine restart, coupled AEC/NS resolution, plus the
reported
sequence replayed end to end.
- `flutter test` — 515 tests (+4), covering the capture-options map that
decides
this natively.
- New `native-test-check` CI job on `macos-latest`: no simulator, device
or pod
install, so it gates every PR.
## Known limitations
- **Per-capture `AudioProcessingMode.software` is still ignored by the
audio
device module.** Verified on device: `isPlatformActive: true`,
`isSoftwareActive: false`. Only the policy above reliably keeps VPIO
away. The
authoritative fix belongs in the webrtc-sdk fork — the ADM should honour
the
request or reject it with a clear error. Documented on the API for now.
A visible consequence: when the ADM ignores a software request, the
session
policy and the actual processing path disagree (`default` mode with VPIO
active).
- **Routes are not covered.** Measurements are one device, built-in
speaker and
receiver. Bluetooth HFP, wired headsets and mid-call route changes still
need
manual verification — note that `videoChat` restricts the permitted
route set,
so the `default`-mode path allows routes the chat modes do not.
- The sticky `playAndRecord` is the one part that changes behaviour for
every iOS
app with no opt-in: a muted stretch no longer returns to `playback`, so
it loses
`mixWithOthers` and stays on Bluetooth HFP rather than A2DP. It is
separable
from the rest of the fix if we would rather not take that trade now.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>1 parent 4dae650 commit 55ef223
19 files changed
Lines changed: 1343 additions & 12 deletions
File tree
- .changes
- .github/workflows
- ios
- Classes
- livekit_client/Sources/livekit_client
- lib
- src
- audio
- support
- track/local
- macos
- Classes
- livekit_client/Sources/livekit_client
- test
- audio
- native
- Sources/LiveKitAudioSessionPolicy
- Tests/LiveKitAudioSessionPolicyTests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
87 | 100 | | |
88 | 101 | | |
89 | 102 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
459 | 459 | | |
460 | 460 | | |
461 | 461 | | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
462 | 531 | | |
463 | 532 | | |
464 | 533 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
42 | 47 | | |
43 | 48 | | |
44 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
177 | 199 | | |
178 | 200 | | |
179 | 201 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
38 | 51 | | |
39 | 52 | | |
40 | 53 | | |
| |||
96 | 109 | | |
97 | 110 | | |
98 | 111 | | |
99 | | - | |
| 112 | + | |
100 | 113 | | |
101 | 114 | | |
102 | 115 | | |
| |||
0 commit comments