Add Romanian voice cloning via a new F5-TTS engine - #972
Open
at-aepeak-com wants to merge 15 commits into
Open
Conversation
New 'mms' engine wrapping facebook/mms-tts-ron (Meta MMS, VITS architecture) — pure PyTorch via the already-pinned transformers dependency, zero new packages. Single preset voice, no cloning, modeled on the Kokoro preset-voice pattern. The mms-tts-ron vocab mixes Romanian diacritic conventions (comma-below s U+0219 but cedilla t U+0163), and the character-level VitsTokenizer silently drops out-of-vocab characters, so input text is NFC-normalized and both real-world variants are mapped onto the trained forms before tokenizing. Registered in the model config registry, engine factory, request-model regexes (language 'ro', engine 'mms'), preset-voice endpoints, and the PyInstaller hidden imports. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Romanian added to ALL_LANGUAGES, 'mms' wired through every engine enumeration: engine union types, zod schema, engine picker, display names, preset-engine sets (profile form/list/card, floating generate box), model management filter, and captures play-as union. MMS is treated as a preset-voice engine like Kokoro and Qwen CustomVoice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit coverage for the Romanian diacritics normalization (cedilla vs comma-below variants, NFC composition, lossless handling of unknown chars), engine registration surfaces, and request-model regexes. End-to-end generation tests (model download ~150MB) are opt-in via VOICEBOX_MMS_E2E=1: Romanian text with both diacritic conventions produces valid 16kHz float32 audio, both spellings tokenize identically, seeded generation is deterministic, and fully out-of-vocab text falls back to silence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README engine/language counts (7 to 8 engines, 23 to 24 languages) and tables, preset-voices docs page with the MMS section, PROJECT_STATUS engine tracking, and a CHANGELOG entry under Unreleased. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CapturesTab: only honor a capture's language when the target engine supports it (MMS is Romanian-only; also hardens Kokoro playback), falling back to the profile's language otherwise - models.py: extract shared TTS_LANGUAGE_PATTERN / TTS_ENGINE_PATTERN constants to replace seven duplicated inline regexes - mms_backend: correct module docstring — adding a language also needs the checkpoint resolution threaded through model loading - README: mention MMS (Romanian) in the preset-voices bullet Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wraps the community Romanian F5-TTS fine-tune (MihaiPopa-1/F5-TTS-Romanian, Apache-2.0) for zero-shot voice cloning — the first engine with true Romanian cloning. Follows the Chatterbox cloning pattern: the voice prompt stores the reference audio path + transcript, processed at generation time. - References over 12s are trimmed at the quietest 300ms window between 8-12s (F5 conditioning degrades with long refs; upstream hard-clips at 12s), with the transcript proportionally truncated to keep correspondence and avoid f5-tts's Whisper auto-transcription download. - Romanian diacritics are normalized to the comma-below forms: the fine-tune's vocab lacks cedilla ţ (U+0163) and f5-tts silently maps out-of-vocab chars to space. - MPS enabled: verified stable on this checkpoint with memory free and ~2x faster than CPU (65.6s vs 125.8s for 5.5s of audio). - Cached-state check covers both the fine-tune repo and the Vocos vocoder that f5-tts downloads on init. - New dependency: f5-tts==1.1.21 (pinned), imported lazily. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- TTS_ENGINES entry, ModelConfig (f5-tts-romanian, ro+en), and factory branch in backends/__init__.py - f5 added to TTS_ENGINE_PATTERN (covers all validation sites) - f5 added to CLONING_ENGINES so cloned profiles accept it; no preset branches — F5 is a cloning engine - PyInstaller: hidden-import + collect-all f5_tts (yaml model configs are data files read via importlib.resources); excluded from the MCP shim Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cloning-engine treatment (like Chatterbox), not preset: added to the engine unions, zod enum, model-name/display chains, engine selector (+ CLONING_ENGINES set), cloned-profile default-engine options, model management filter, and ENGINE_LANGUAGES (ro, en). Deliberately NOT in PRESET_ONLY_ENGINES / PRESET_ENGINES / preset badge maps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit coverage: diacritics normalization (cedilla -> comma-below, the reverse direction from MMS), vocab coverage pinning (skips when the checkpoint isn't cached), reference trimming (cut lands in a synthetic silence gap, never exceeds 12s, proportional text truncation), engine registration surfaces, and prompt validation (missing ref audio and empty ref text are refused before the model loads). E2E gated behind VOICEBOX_F5_E2E=1: generates the Romanian reference in-test with the MMS engine, clones through the real checkpoint on MPS, asserts 24kHz float32 non-NaN output and seeded determinism (passed in 96s on this machine). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README engine counts (8 to 9) and tables — Romanian now has true voice cloning, so the MMS row drops the 'only engine with Romanian' claim. PROJECT_STATUS engine tracking and model matrix, and a CHANGELOG entry under Unreleased that flags the new f5-tts dependency, the FFmpeg system requirement, and the honest speed caveat (~12x realtime on MPS, ~20x on CPU). preset-voices.mdx is untouched — F5 is a cloning engine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds Romanian MMS preset TTS and F5 voice cloning across backend registration, inference, validation, frontend selection, tests, packaging, documentation, and release notes. ChangesRomanian TTS engine integration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant GenerationForm
participant BackendFactory
participant MMSTTSBackend
participant F5TTSBackend
participant ModelCheckpoint
GenerationForm->>BackendFactory: submit mms or f5 generation request
BackendFactory->>MMSTTSBackend: create mms backend
BackendFactory->>F5TTSBackend: create f5 backend
MMSTTSBackend->>ModelCheckpoint: load MMS tokenizer and model
F5TTSBackend->>ModelCheckpoint: load F5 checkpoint and vocoder
MMSTTSBackend-->>GenerationForm: return preset Romanian audio
F5TTSBackend-->>GenerationForm: return cloned Romanian audio
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/components/CapturesTab/CapturesTab.tsx`:
- Around line 269-282: Update the engine selection in the capture playback path
to use the effective profile engine, falling back from voice.default_engine to
voice.preset_engine as FloatingGenerateBox does. Use this resolved engine for
both the request payload and the ENGINE_LANGUAGES compatibility check so
preset-only profiles, including MMS, retain their configured engine and language
handling.
In `@CHANGELOG.md`:
- Around line 20-21: Update the README Quick Start prerequisites to explicitly
list FFmpeg as a required system dependency for F5 generation and torchcodec
audio I/O. Keep the existing dependency instructions intact and ensure FFmpeg
appears in the installation/setup documentation, not only the changelog.
In `@docs/content/docs/overview/preset-voices.mdx`:
- Around line 10-16: Update the MMS row and its related description around the
preset-voices overview to say MMS is the only preset engine with Romanian, not
the only engine in the app. Preserve the existing Romanian availability and
performance details while changing both conflicting claims.
- Around line 12-15: Update the Kokoro 82M language count from 9 to 8 in the
preset-voices table and its related section heading, preserving the listed
supported languages and all other documentation content.
In `@docs/PROJECT_STATUS.md`:
- Around line 267-271: Update the documented generation flow’s engine-resolution
list to include both mms and f5, matching the engine union shown in the
Multi-Engine Architecture section. Preserve all existing engine identifiers and
ordering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 328e7061-f748-45e2-9107-63c32fc6db0f
📒 Files selected for processing (25)
CHANGELOG.mdREADME.mdapp/src/components/CapturesTab/CapturesTab.tsxapp/src/components/Generation/EngineModelSelector.tsxapp/src/components/Generation/FloatingGenerateBox.tsxapp/src/components/ServerSettings/ModelManagement.tsxapp/src/components/VoiceProfiles/ProfileCard.tsxapp/src/components/VoiceProfiles/ProfileForm.tsxapp/src/components/VoiceProfiles/ProfileList.tsxapp/src/lib/api/types.tsapp/src/lib/constants/languages.tsapp/src/lib/hooks/useGenerationForm.tsapp/src/lib/utils/format.tsbackend/backends/__init__.pybackend/backends/f5_backend.pybackend/backends/mms_backend.pybackend/build_binary.pybackend/models.pybackend/requirements.txtbackend/routes/profiles.pybackend/services/profiles.pybackend/tests/test_f5_backend.pybackend/tests/test_mms_backend.pydocs/PROJECT_STATUS.mddocs/content/docs/overview/preset-voices.mdx
Comment on lines
267
to
+271
| ### Multi-Engine Architecture (Shipped) | ||
|
|
||
| - **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking | ||
| - **Per-engine backend instances** — each engine gets its own singleton, loaded lazily | ||
| - **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro'` | ||
| - **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro' | 'mms' | 'f5'` |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Keep the documented generation flow in sync with the new engine union.
The new contract includes mms and f5, but the flow at Lines 76-77 still says engine resolution handles only through kokoro. Add both identifiers there so the architecture documentation does not contradict the shipped contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/PROJECT_STATUS.md` around lines 267 - 271, Update the documented
generation flow’s engine-resolution list to include both mms and f5, matching
the engine union shown in the Multi-Engine Architecture section. Preserve all
existing engine identifiers and ordering.
- CapturesTab: resolve effective engine as default_engine ?? preset_engine (matches FloatingGenerateBox) so preset-only profiles keep their engine and language guard during capture playback - README: add FFmpeg to Quick Start prerequisites - preset-voices.mdx: MMS is the only *preset* engine with Romanian (F5 clones it); fix Kokoro language count 9 -> 8 - PROJECT_STATUS: add mms | f5 to the documented engine resolution flow Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets users point the f5 engine at a personal fine-tuned checkpoint (pruned safetensors) without code changes; vocab optionally overridable via VOICEBOX_F5_VOCAB. Missing paths log a warning and fall back to the HF checkpoint. Unit-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/backends/f5_backend.py`:
- Around line 42-47: Validate VOICEBOX_F5_VOCAB through a new _vocab_override()
helper mirroring _ckpt_override(), returning the override only when the path
exists and otherwise warning and falling back to the Hugging Face vocabulary.
Replace direct vocabulary environment-variable usage with this helper, and
update _is_model_cached() to recognize valid local vocabulary overrides; add
tests covering both missing and existing override paths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: de7b7a52-5dbc-4619-859c-43ac48b352ea
📒 Files selected for processing (2)
backend/backends/f5_backend.pybackend/tests/test_f5_backend.py
Comment on lines
+42
to
+47
| # Local checkpoint override for personal fine-tunes: point VOICEBOX_F5_CKPT at | ||
| # a pruned .safetensors (and optionally VOICEBOX_F5_VOCAB at a matching | ||
| # vocab.txt — defaults to the repo vocab, which personal fine-tunes based on | ||
| # it share). When set, the checkpoint download is skipped entirely. | ||
| F5_CKPT_OVERRIDE_ENV = "VOICEBOX_F5_CKPT" | ||
| F5_VOCAB_OVERRIDE_ENV = "VOICEBOX_F5_VOCAB" |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Validate the vocabulary override before using it.
Unlike the checkpoint override, VOICEBOX_F5_VOCAB is passed through without checking whether the file exists. A typo or stale path therefore causes model loading to fail instead of warning and falling back to the Hugging Face vocabulary. The cache check also ignores a valid local vocabulary override.
Add a _vocab_override() helper mirroring _ckpt_override(), use it here and in _is_model_cached(), and add missing/existing-path tests.
Proposed direction
+ `@staticmethod`
+ def _vocab_override() -> str | None:
+ path = os.environ.get(F5_VOCAB_OVERRIDE_ENV)
+ if path and Path(path).is_file():
+ return path
+ if path:
+ logger.warning(
+ "%s=%s does not exist — falling back to the HF vocabulary",
+ F5_VOCAB_OVERRIDE_ENV,
+ path,
+ )
+ return None
+
- vocab_file = os.environ.get(F5_VOCAB_OVERRIDE_ENV) or hf_hub_download(F5_HF_REPO, F5_VOCAB_FILE)
+ vocab_file = self._vocab_override() or hf_hub_download(F5_HF_REPO, F5_VOCAB_FILE)Also applies to: 183-190, 210-215
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/backends/f5_backend.py` around lines 42 - 47, Validate
VOICEBOX_F5_VOCAB through a new _vocab_override() helper mirroring
_ckpt_override(), returning the override only when the path exists and otherwise
warning and falling back to the Hugging Face vocabulary. Replace direct
vocabulary environment-variable usage with this helper, and update
_is_model_cached() to recognize valid local vocabulary overrides; add tests
covering both missing and existing override paths.
Digit characters are in the fine-tune's vocab but effectively untrained (the training data writes numbers in letters), so raw digits came out garbled. Romanian text now gets numbers spelled out before synthesis: cardinals with gender agreement and the "de" linker, decimal commas, clock times and percentages. Also adds VOICEBOX_F5_SPEED (0.3-2.0, default 1.0) to compensate fine-tunes whose training data was read faster than the desired output pace. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GenerationRequest.language defaulted to "en", so API callers that omitted it silently bypassed language-specific normalization (e.g. Romanian diacritic and number handling) even on profiles configured for another language. The field is now optional and falls back to the profile's language in both the generate and stream routes; an explicit request value still wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/backends/f5_backend.py`:
- Around line 148-160: Update the number-formatting loop in the function
containing _RO_SCALES so groups above 999 are handled before calling
_ro_under_1000: extend the Romanian scale definitions if appropriate, or
preserve the original numeric token for unsupported magnitudes without raising.
Add a regression test covering 1_000_000_000_000 and verify formatting no longer
crashes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 2f50416b-1d3e-4a2a-9455-5c08e3efc470
📒 Files selected for processing (4)
backend/backends/f5_backend.pybackend/models.pybackend/routes/generations.pybackend/tests/test_f5_backend.py
Comment on lines
+148
to
+160
| for value, singular, plural in _RO_SCALES: | ||
| group, n = divmod(n, value) | ||
| if not group: | ||
| continue | ||
| if group == 1: | ||
| parts.append(singular) | ||
| else: | ||
| # groups counting mii/milioane are grammatically feminine, and | ||
| # 20+ links with "de": "douăzeci de mii", but "douăsprezece mii" | ||
| link = " " if 1 <= group % 100 <= 19 else " de " | ||
| parts.append(_ro_under_1000(group, feminine=True) + link + plural) | ||
| if n: | ||
| parts.append(_ro_under_1000(n, feminine)) |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Prevent crashes for numbers above the supported scale.
_RO_SCALES ends at 1_000_000_000. For 1_000_000_000_000, group is 1000, and line 158 passes it to _ro_under_1000. That function accesses _RO_UNITS[10] and raises IndexError.
Handle groups above 999 before calling _ro_under_1000. Extend scale support or preserve unsupported numeric tokens. Add a regression test for 1_000_000_000_000.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/backends/f5_backend.py` around lines 148 - 160, Update the
number-formatting loop in the function containing _RO_SCALES so groups above 999
are handled before calling _ro_under_1000: extend the Romanian scale definitions
if appropriate, or preserve the original numeric token for unsupported
magnitudes without raising. Add a regression test covering 1_000_000_000_000 and
verify formatting no longer crashes.
- Fix a hard crash on long/chunked F5 generation: F5TTS.infer runs
seed_everything(seed), which writes PYTHONHASHSEED; with seed=None F5
drew random.randint(0, sys.maxsize) (~9e18), so the next spawned
subprocess aborted with "PYTHONHASHSEED must be in range
[0, 4294967295]". Always pass F5 a seed inside that range.
- Chunk F5 output to ~one sentence (VOICEBOX_F5 default 140 chars):
F5 loses coherence generating long single-shot audio (>~15s came out
slurred). max_chunk_chars is now per-engine (None -> engine default).
- Add opt-in generation knobs on the F5 engine:
* VOICEBOX_F5_NFE (16-128, default 32) - flow-matching steps.
* VOICEBOX_F5_BEST_OF (1-8, default 1) - generate N candidates and keep
the one an ASR pass transcribes closest to the intended text; the
scorer spells numbers in the ASR output so the digit-vs-words format
gap can't flatten every candidate to the same score.
* VOICEBOX_F5_ONSET_FIX (default off) - F5 garbles a leading t-comma /
i-circumflex sound (measured; vowel/common onsets are clean); a
throwaway lead-in word absorbs it. Off by default because trimming the
lead-in back off via ASR timestamps can clip real speech.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
backend/backends/f5_backend.py (2)
639-647: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider tolerating a scoring failure inside the best-of-N loop.
_score_candidateat Line 642 has no error handling. It loads a Whisper pipeline on first use, which can fail on a download error or an out-of-memory condition. The exception then propagates and the whole generation fails, even though a usable candidate is already in hand.
_trim_lead_inalready applies the opposite policy and degrades gracefully. Catch the exception per candidate, log it, and score that candidate as0.0so the loop still returns audio.♻️ Proposed refactor
for i in range(best_of): cand_seed = None if seed is None else seed + i audio, sr = _infer_once(cand_seed) - score = self._score_candidate(audio, sr, gen_text) + try: + score = self._score_candidate(audio, sr, gen_text) + except Exception as e: + logger.warning("[F5] candidate scoring failed, scoring 0: %s", e) + score = 0.0 logger.info("[F5] best-of-%d candidate %d/%d score=%.3f", best_of, i + 1, best_of, score)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/backends/f5_backend.py` around lines 639 - 647, Update the best-of-N loop around _score_candidate to catch per-candidate scoring exceptions, log the failure with relevant context, and assign that candidate a score of 0.0 so iteration continues and usable audio can still be selected.
381-396: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winNormalize diacritics when matching the lead-in word.
lead_wordsholds"așa"with the comma-below diacritic. Whisper often transcribes Romanian without diacritics or with the cedilla form, so it can return"asa"or"aşa". The membership test at Line 396 then fails on the first chunk, the code treats the lead-in as real speech, and the trim is skipped. The throwaway word stays in the output.
_asr_similarity_keyalready strips diacritics and punctuation. Reuse it on both sides.The feature is opt-in and the comments already record the trim as imprecise, so this is a follow-up rather than a blocker.
♻️ Proposed refactor
- lead_words = {w.strip(".,!?").lower() for w in lead_in.split() if w.strip(".,!?")} + lead_words = {k for k in (_asr_similarity_key(w) for w in lead_in.split()) if k} try: @@ for chunk in result.get("chunks", []): - word = chunk["text"].strip().strip(".,!?").lower() + word = _asr_similarity_key(chunk["text"]) if word and word not in lead_words:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/backends/f5_backend.py` around lines 381 - 396, Normalize both the lead-in words and each Whisper-transcribed chunk with the existing _asr_similarity_key before the membership comparison, while preserving the current punctuation trimming and opt-in behavior. Update the lead_words construction and the word-matching logic in the ASR scoring flow; do not introduce a separate diacritic-normalization implementation.backend/tests/test_f5_backend.py (1)
179-209: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd boundary values to the range tests.
The invalid sets cover below-range, above-range, non-numeric, and empty input. The accepted bounds themselves are untested.
_env_intuseslo <= value <= hi, so a change to<would keep every current test green while rejecting 16, 128, 1, and 8.Parametrize the valid cases with the inclusive limits.
♻️ Proposed refactor
- def test_nfe_valid(self, monkeypatch): + `@pytest.mark.parametrize`("raw,expected", [("64", 64), ("16", 16), ("128", 128)]) + def test_nfe_valid(self, monkeypatch, raw, expected): from backend.backends.f5_backend import F5_NFE_ENV, _f5_nfe_steps - monkeypatch.setenv(F5_NFE_ENV, "64") - assert _f5_nfe_steps() == 64 + monkeypatch.setenv(F5_NFE_ENV, raw) + assert _f5_nfe_steps() == expected @@ - def test_best_of_valid(self, monkeypatch): + `@pytest.mark.parametrize`("raw,expected", [("3", 3), ("1", 1), ("8", 8)]) + def test_best_of_valid(self, monkeypatch, raw, expected): from backend.backends.f5_backend import F5_BEST_OF_ENV, _f5_best_of - monkeypatch.setenv(F5_BEST_OF_ENV, "3") - assert _f5_best_of() == 3 + monkeypatch.setenv(F5_BEST_OF_ENV, raw) + assert _f5_best_of() == expected🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/tests/test_f5_backend.py` around lines 179 - 209, Extend the valid-case parametrization in the tests for _f5_nfe_steps and _f5_best_of to include both inclusive boundary values: 16 and 128 for NFE, and 1 and 8 for best-of. Keep the existing valid interior cases and assertions, verifying each boundary is accepted by the inclusive range logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/backends/f5_backend.py`:
- Around line 346-359: Unify lazy Whisper pipeline creation by adding a
_get_scorer() helper that creates self._scorer with device="cpu", then update
both _score_candidate and _trim_lead_in to use it instead of creating or
initializing the scorer independently. Update unload_model to delete and reset
self._scorer before or alongside releasing self.model.
---
Nitpick comments:
In `@backend/backends/f5_backend.py`:
- Around line 639-647: Update the best-of-N loop around _score_candidate to
catch per-candidate scoring exceptions, log the failure with relevant context,
and assign that candidate a score of 0.0 so iteration continues and usable audio
can still be selected.
- Around line 381-396: Normalize both the lead-in words and each
Whisper-transcribed chunk with the existing _asr_similarity_key before the
membership comparison, while preserving the current punctuation trimming and
opt-in behavior. Update the lead_words construction and the word-matching logic
in the ASR scoring flow; do not introduce a separate diacritic-normalization
implementation.
In `@backend/tests/test_f5_backend.py`:
- Around line 179-209: Extend the valid-case parametrization in the tests for
_f5_nfe_steps and _f5_best_of to include both inclusive boundary values: 16 and
128 for NFE, and 1 and 8 for best-of. Keep the existing valid interior cases and
assertions, verifying each boundary is accepted by the inclusive range logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a8a140a5-0fa4-4a48-8da2-ef388c0ec2a1
📒 Files selected for processing (5)
backend/backends/__init__.pybackend/backends/f5_backend.pybackend/models.pybackend/routes/generations.pybackend/tests/test_f5_backend.py
🚧 Files skipped from review as they are similar to previous changes (2)
- backend/backends/init.py
- backend/models.py
Comment on lines
+346
to
+359
| self._scorer = None # lazy Whisper pipeline for best-of-N ranking | ||
|
|
||
| def _score_candidate(self, audio: np.ndarray, sample_rate: int, target: str) -> float: | ||
| """Similarity in [0, 1] between an ASR transcription of `audio` and | ||
| the intended text. Used only when best-of-N is enabled.""" | ||
| from difflib import SequenceMatcher | ||
|
|
||
| if self._scorer is None: | ||
| from transformers import pipeline as hf_pipeline | ||
|
|
||
| logger.info("[F5] Loading ASR scorer for best-of-N (whisper-small)...") | ||
| self._scorer = hf_pipeline( | ||
| "automatic-speech-recognition", model="openai/whisper-small" | ||
| ) |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Unify the lazy _scorer creation and release it on unload.
_score_candidate and _trim_lead_in both create self._scorer, but with different arguments. _score_candidate omits device="cpu". _trim_lead_in sets it, and its docstring states the ASR must run on CPU to avoid contending with F5 on MPS.
The two helpers share one attribute, so the first caller fixes the device for the second. If VOICEBOX_F5_BEST_OF is above 1 and the onset fix is enabled, _score_candidate runs first at Line 642 and _trim_lead_in reuses its scorer at Line 649. The documented CPU placement is then never applied.
unload_model also clears self.model but leaves self._scorer resident, so the Whisper pipeline keeps its memory after an unload.
Extract a single _get_scorer() helper with one device choice, and drop the scorer in unload_model.
♻️ Proposed refactor
+ def _get_scorer(self):
+ """Lazy Whisper pipeline, pinned to CPU so it never contends with
+ F5 on MPS. Shared by best-of-N scoring and onset lead-in trimming."""
+ if self._scorer is None:
+ from transformers import pipeline as hf_pipeline
+
+ logger.info("[F5] Loading ASR helper (whisper-small) on CPU...")
+ self._scorer = hf_pipeline(
+ "automatic-speech-recognition", model="openai/whisper-small", device="cpu"
+ )
+ return self._scorer
+
def _score_candidate(self, audio: np.ndarray, sample_rate: int, target: str) -> float:
"""Similarity in [0, 1] between an ASR transcription of `audio` and
the intended text. Used only when best-of-N is enabled."""
from difflib import SequenceMatcher
- if self._scorer is None:
- from transformers import pipeline as hf_pipeline
-
- logger.info("[F5] Loading ASR scorer for best-of-N (whisper-small)...")
- self._scorer = hf_pipeline(
- "automatic-speech-recognition", model="openai/whisper-small"
- )
- heard = self._scorer(
+ heard = self._get_scorer()(
{"array": np.asarray(audio, dtype=np.float32), "sampling_rate": sample_rate},
generate_kwargs={"language": "romanian", "task": "transcribe"},
)["text"]Apply the same helper in _trim_lead_in, and release the scorer in unload_model:
def unload_model(self) -> None:
"""Unload model to free memory."""
if self._scorer is not None:
del self._scorer
self._scorer = None
if self.model is not None:
...🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/backends/f5_backend.py` around lines 346 - 359, Unify lazy Whisper
pipeline creation by adding a _get_scorer() helper that creates self._scorer
with device="cpu", then update both _score_candidate and _trim_lead_in to use it
instead of creating or initializing the scorer independently. Update
unload_model to delete and reset self._scorer before or alongside releasing
self.model.
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Second Romanian engine, complementing #916: a new
f5TTS engine wrapping the community Romanian F5-TTS fine-tune (MihaiPopa-1/F5-TTS-Romanian, Apache-2.0, baseSWivid/F5-TTS) — the first engine with true Romanian voice cloning: zero-shot cloning from a short reference sample, speaking Romanian with native phonology.Why
#916 gives Romanian a preset voice; this gives Romanian cloning — users can speak Romanian in their own voice. MMS structurally cannot clone (single-speaker VITS); F5 conditions on reference audio by design.
Key implementation notes
chatterbox_backend.py(reference audio + text processed at generation time), not the preset pattern.f5-tts==1.1.21(pinned, imported lazily). FFmpeg becomes a system requirement (torchaudio ≥2.11 I/O). Called out in docs.build_binary.py:--collect-all f5_tts(its yaml model configs load viaimportlib.resources; a hidden-import alone won't bundle them).Testing
VOICEBOX_F5_E2E=1(real checkpoint download, cloning from an MMS-generated Romanian reference, 24 kHz float32 output, seeded determinism — exact match on MPS).f5→ Romanian generation; Model Management download/load/unload truthful (checkpoint + vocab + Vocos vocoder all checked).ruffclean on new files;tsc --noEmitclean; no new biome findings on touched files.Upgrade path
When RACAI publishes official Ro-F5TTS weights (trained on 21h SWARA vs this checkpoint's 1.3h Common Voice — paper), this engine upgrades by swapping
F5_HF_REPO/F5_CKPT_FILE— no architectural change.🤖 Generated with Claude Code
Summary by CodeRabbit