Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,27 @@

## [Unreleased]

### Voice Generation

- **Romanian voice cloning via a new F5-TTS engine.** A new `f5` engine wraps
the community Romanian fine-tune of F5-TTS (`MihaiPopa-1/F5-TTS-Romanian`,
Apache-2.0) — the first engine with true Romanian voice cloning: record a
cloned profile and generate Romanian (or English) speech in that voice,
zero-shot. References longer than 12s are trimmed at the quietest pause and
both Romanian diacritic conventions are normalized to the model's vocab.
Heads-up: generation uses 32-step flow matching and is well below realtime
(~12x slower on Apple Silicon MPS, ~20x on CPU); the ~1.2GB checkpoint plus
the Vocos vocoder download on first use. Adds one Python dependency
(`f5-tts`, pinned); FFmpeg is required on the system (torchcodec audio I/O).
Comment thread
coderabbitai[bot] marked this conversation as resolved.

- **Romanian TTS via a new MMS engine.** A new `mms` engine wraps Meta's
MMS-TTS Romanian checkpoint (`facebook/mms-tts-ron`) — a ~150MB VITS model
that runs realtime on CPU, using the already-bundled transformers runtime
(zero new dependencies). It ships as a preset voice (like Kokoro): create a
profile with the Romanian MMS voice and generate. Romanian text using either
diacritic convention (cedilla ş/ţ or comma-below ș/ț) is normalized
automatically so no character is silently dropped by the tokenizer.

### Linux

- **ROCm setup works on Linux AMD systems.** Docker ROCm builds now keep PyTorch
Expand Down
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,14 @@

## What is Voicebox?

Voicebox is a **local-first AI voice studio** — a free and open-source alternative to **ElevenLabs** and **WisprFlow** in one app. Clone voices from a few seconds of audio, generate speech in 23 languages across 7 TTS engines, dictate into any text field with a global hotkey, and give any MCP-aware AI agent a voice of your choosing.
Voicebox is a **local-first AI voice studio** — a free and open-source alternative to **ElevenLabs** and **WisprFlow** in one app. Clone voices from a few seconds of audio, generate speech in 24 languages across 9 TTS engines, dictate into any text field with a global hotkey, and give any MCP-aware AI agent a voice of your choosing.

The two cloud incumbents sit on opposite halves of the voice I/O loop — ElevenLabs on output, WisprFlow on input. Voicebox does both, bridges them with a bundled local LLM for refinement and per-profile personas, and runs the whole thing on your machine.

- **Complete privacy** — models, voice data, and captures never leave your machine
- **7 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro
- **Voice cloning and preset voices** — zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro and Qwen CustomVoice
- **23 languages** — from English to Arabic, Japanese, Hindi, Swahili, and more
- **9 TTS engines** — Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, Kokoro, MMS, and F5-TTS
- **Voice cloning and preset voices** — zero-shot cloning from a reference sample, or 50+ curated preset voices via Kokoro, Qwen CustomVoice, and MMS (Romanian)
- **24 languages** — from English to Arabic, Japanese, Hindi, Romanian, Swahili, and more
- **Post-processing effects** — pitch shift, reverb, delay, chorus, compression, and filters
- **Expressive speech** — paralinguistic tags like `[laugh]`, `[sigh]`, `[gasp]` via Chatterbox Turbo; natural-language delivery control via Qwen CustomVoice
- **Unlimited length** — auto-chunking with crossfade for scripts, articles, and chapters
Expand Down Expand Up @@ -109,7 +109,7 @@ The two cloud incumbents sit on opposite halves of the voice I/O loop — Eleven

### Multi-Engine Voice Cloning

Seven TTS engines with different strengths, switchable per-generation:
Nine TTS engines with different strengths, switchable per-generation:

| Engine | Languages | Strengths |
| --------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
Expand All @@ -120,6 +120,8 @@ Seven TTS engines with different strengths, switchable per-generation:
| **Chatterbox Turbo** | English | Fast 350M model with paralinguistic emotion/sound tags |
| **TADA** (1B / 3B) | 10 | HumeAI speech-language model — 700s+ coherent audio, text-acoustic dual alignment |
| **Kokoro** | 8 | 50 curated preset voices, tiny 82M model, fast CPU inference |
| **MMS** (Meta) | Romanian | Per-language VITS checkpoints (~150MB), CPU realtime, preset voice |
| **F5-TTS Romanian** | Romanian, English | Community fine-tune with true Romanian voice cloning — zero-shot from a reference sample (slow: flow matching, ~12-20x realtime) |

### Emotions & Paralinguistic Tags

Expand Down Expand Up @@ -369,7 +371,7 @@ Full API documentation available at `http://127.0.0.1:17493/docs`.
| Frontend | React, TypeScript, Tailwind CSS |
| State | Zustand, React Query |
| Backend | FastAPI (Python) |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro |
| TTS Engines | Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Kokoro, MMS, F5-TTS |
| STT | Whisper / Whisper Turbo (PyTorch or MLX) |
| Local LLM | Qwen3 (0.6B / 1.7B / 4B), shared runtime with TTS / STT |
| MCP Server | FastMCP mounted at `/mcp` (Streamable HTTP) + bundled stdio shim binary |
Expand Down Expand Up @@ -416,7 +418,7 @@ just dev # starts backend + desktop app

Install [just](https://github.com/casey/just): `brew install just` or `cargo install just`. Run `just --list` to see all commands.

**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org), [Tauri Prerequisites](https://v2.tauri.app/start/prerequisites/), and [Xcode](https://developer.apple.com/xcode/) on macOS.
**Prerequisites:** [Bun](https://bun.sh), [Rust](https://rustup.rs), [Python 3.11+](https://python.org), [FFmpeg](https://ffmpeg.org) (`brew install ffmpeg` — required for F5-TTS generation and torchaudio ≥2.11 audio I/O), [Tauri Prerequisites](https://v2.tauri.app/start/prerequisites/), and [Xcode](https://developer.apple.com/xcode/) on macOS.

The repo ships a pre-wired `.mcp.json` at the root — running Claude Code inside this checkout picks up the Voicebox MCP tools automatically once the dev app is running.

Expand Down
17 changes: 13 additions & 4 deletions app/src/components/CapturesTab/CapturesTab.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ import type {
CaptureSource,
VoiceProfileResponse,
} from '@/lib/api/types';
import type { LanguageCode } from '@/lib/constants/languages';
import { ENGINE_LANGUAGES, type LanguageCode } from '@/lib/constants/languages';
import { BOTTOM_SAFE_AREA_PADDING } from '@/lib/constants/ui';
import { useCaptureRecordingSession } from '@/lib/hooks/useCaptureRecordingSession';
import { useDictationReadiness } from '@/lib/hooks/useDictationReadiness';
Expand Down Expand Up @@ -263,14 +263,23 @@ export function CapturesTab() {
mutationFn: async ({ capture, voice }: { capture: CaptureResponse; voice: VoiceProfileResponse }) => {
const text = capture.transcript_refined || capture.transcript_raw;
if (!text.trim()) throw new Error(t('captures.noTranscriptError'));
const language = (capture.language || voice.language) as LanguageCode;
// Preset profiles (Kokoro etc.) reject the qwen default — honor the
// profile's stored engine preference. Cloned profiles without an
// override fall through to whatever the backend picks.
const engine = voice.default_engine as
const engine = (voice.default_engine ?? voice.preset_engine) as
| 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox'
| 'chatterbox_turbo' | 'tada' | 'kokoro'
| 'chatterbox_turbo' | 'tada' | 'kokoro' | 'mms' | 'f5'
| undefined;
// Prefer the capture's language, but only if the target engine can
// speak it (e.g. MMS is Romanian-only, Kokoro covers 8 languages) —
// otherwise fall back to the profile's own language.
const captureLanguage = capture.language as LanguageCode | undefined;
const supported = engine ? ENGINE_LANGUAGES[engine] : undefined;
const language = (
captureLanguage && (!supported || supported.includes(captureLanguage))
? captureLanguage
: voice.language
) as LanguageCode;
return apiClient.generateSpeech({
profile_id: voice.id,
text,
Expand Down
6 changes: 5 additions & 1 deletion app/src/components/Generation/EngineModelSelector.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ const ENGINE_OPTIONS = [
{ value: 'tada:1B', label: 'TADA 1B', engine: 'tada' },
{ value: 'tada:3B', label: 'TADA 3B Multilingual', engine: 'tada' },
{ value: 'kokoro', label: 'Kokoro 82M', engine: 'kokoro' },
{ value: 'mms', label: 'MMS Romanian', engine: 'mms' },
{ value: 'f5', label: 'F5-TTS Romanian', engine: 'f5' },
] as const;

const ENGINE_DESCRIPTIONS: Record<string, string> = {
Expand All @@ -37,13 +39,15 @@ const ENGINE_DESCRIPTIONS: Record<string, string> = {
chatterbox_turbo: 'English, [laugh] [cough] tags',
tada: 'HumeAI, 700s+ coherent audio',
kokoro: '82M params, CPU realtime, 8 langs',
mms: 'Meta MMS, Romanian, CPU realtime',
f5: 'Romanian voice cloning, slow but faithful',
};

/** Engines that only support English and should force language to 'en' on select. */
const ENGLISH_ONLY_ENGINES = new Set(['luxtts', 'chatterbox_turbo']);

/** Engines that support cloned (reference audio) profiles. */
const CLONING_ENGINES = new Set(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada']);
const CLONING_ENGINES = new Set(['qwen', 'luxtts', 'chatterbox', 'chatterbox_turbo', 'tada', 'f5']);

function getAvailableOptions(selectedProfile?: VoiceProfileResponse | null) {
if (!selectedProfile) return ENGINE_OPTIONS;
Expand Down
17 changes: 12 additions & 5 deletions app/src/components/Generation/FloatingGenerateBox.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,9 @@ export function FloatingGenerateBox({
| 'chatterbox_turbo'
| 'tada'
| 'kokoro'
| 'qwen_custom_voice';
| 'qwen_custom_voice'
| 'mms'
| 'f5';
useEffect(() => {
if (selectedProfile?.language) {
form.setValue('language', selectedProfile.language as LanguageCode);
Expand All @@ -163,7 +165,7 @@ export function FloatingGenerateBox({
} else if (selectedProfile && selectedProfile.voice_type !== 'preset') {
// Cloned/designed profile with no default — ensure a compatible (non-preset) engine
const currentEngine = form.getValues('engine');
const presetEngines = new Set(['kokoro', 'qwen_custom_voice']);
const presetEngines = new Set(['kokoro', 'qwen_custom_voice', 'mms']);
if (currentEngine && presetEngines.has(currentEngine)) {
form.setValue('engine', 'qwen');
}
Expand Down Expand Up @@ -418,13 +420,19 @@ export function FloatingGenerateBox({
? 'bg-accent text-accent-foreground border border-accent hover:bg-accent/90'
: 'bg-card border border-border hover:bg-background/50',
)}
aria-label={active ? t('generation.persona.ariaLabelActive') : t('generation.persona.ariaLabelInactive')}
aria-label={
active
? t('generation.persona.ariaLabelActive')
: t('generation.persona.ariaLabelInactive')
}
aria-pressed={active}
>
<Wand2 className="h-4 w-4" />
</Button>
<span className="pointer-events-none absolute bottom-full left-1/2 -translate-x-1/2 mb-2 whitespace-nowrap rounded-md bg-popover px-3 py-1.5 text-xs text-popover-foreground border border-border opacity-0 transition-opacity group-hover:opacity-100 z-[9999]">
{active ? t('generation.persona.tooltipActive') : t('generation.persona.tooltipInactive')}
{active
? t('generation.persona.tooltipActive')
: t('generation.persona.tooltipInactive')}
</span>
</div>
</FormControl>
Expand Down Expand Up @@ -566,7 +574,6 @@ export function FloatingGenerateBox({
</div>
)}


<FormField
control={form.control}
name="language"
Expand Down
4 changes: 3 additions & 1 deletion app/src/components/ServerSettings/ModelManagement.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,9 @@ export function ModelManagement() {
m.model_name.startsWith('luxtts') ||
m.model_name.startsWith('chatterbox') ||
m.model_name.startsWith('tada') ||
m.model_name.startsWith('kokoro'),
m.model_name.startsWith('kokoro') ||
m.model_name.startsWith('mms') ||
m.model_name.startsWith('f5'),
) ?? [];
const whisperModels = modelStatus?.models.filter((m) => m.model_name.startsWith('whisper')) ?? [];
const llmModels = modelStatus?.models.filter((m) => m.model_name.startsWith('qwen3-')) ?? [];
Expand Down
1 change: 1 addition & 0 deletions app/src/components/VoiceProfiles/ProfileCard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { useUIStore } from '@/stores/uiStore';
const ENGINE_DISPLAY_NAMES: Record<string, string> = {
kokoro: 'Kokoro',
qwen_custom_voice: 'CustomVoice',
mms: 'MMS',
};

interface ProfileCardProps {
Expand Down
9 changes: 5 additions & 4 deletions app/src/components/VoiceProfiles/ProfileForm.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ import { AudioSampleUpload } from './AudioSampleUpload';
import { SampleList } from './SampleList';

const MAX_AUDIO_DURATION_SECONDS = 30;
const PRESET_ONLY_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
const PRESET_ONLY_ENGINES = new Set(['kokoro', 'qwen_custom_voice', 'mms']);
const DEFAULT_ENGINE_OPTIONS = [
{ value: 'qwen', label: 'Qwen3-TTS' },
{ value: 'qwen_custom_voice', label: 'Qwen CustomVoice' },
Expand All @@ -70,6 +70,8 @@ const DEFAULT_ENGINE_OPTIONS = [
{ value: 'chatterbox_turbo', label: 'Chatterbox Turbo' },
{ value: 'tada', label: 'TADA' },
{ value: 'kokoro', label: 'Kokoro 82M' },
{ value: 'mms', label: 'MMS Romanian' },
{ value: 'f5', label: 'F5-TTS Romanian' },
] as const;

function makeProfileSchema(t: (key: string) => string) {
Expand Down Expand Up @@ -898,6 +900,7 @@ export function ProfileForm() {
<SelectContent>
<SelectItem value="kokoro">Kokoro 82M</SelectItem>
<SelectItem value="qwen_custom_voice">Qwen CustomVoice</SelectItem>
<SelectItem value="mms">MMS Romanian</SelectItem>
</SelectContent>
</Select>
</FormItem>
Expand Down Expand Up @@ -1207,9 +1210,7 @@ export function ProfileForm() {
{...field}
/>
</FormControl>
<FormDescription>
{t('profileForm.fields.personalityHint')}
</FormDescription>
<FormDescription>{t('profileForm.fields.personalityHint')}</FormDescription>
<FormMessage />
</FormItem>
)}
Expand Down
2 changes: 1 addition & 1 deletion app/src/components/VoiceProfiles/ProfileList.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import { ProfileCard } from './ProfileCard';
import { ProfileForm } from './ProfileForm';

/** Engines that use preset (built-in) voices instead of cloned profiles. */
const PRESET_ENGINES = new Set(['kokoro', 'qwen_custom_voice']);
const PRESET_ENGINES = new Set(['kokoro', 'qwen_custom_voice', 'mms']);

export function ProfileList() {
const { t } = useTranslation();
Expand Down
4 changes: 3 additions & 1 deletion app/src/lib/api/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ export interface GenerationRequest {
| 'chatterbox'
| 'chatterbox_turbo'
| 'tada'
| 'kokoro';
| 'kokoro'
| 'mms'
| 'f5';
instruct?: string;
/** When true and the profile has a personality prompt, input text is rewritten in-character before TTS. */
personality?: boolean;
Expand Down
5 changes: 5 additions & 0 deletions app/src/lib/constants/languages.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
* Chatterbox Multilingual supports 23 languages.
* Chatterbox Turbo is English-only.
* Kokoro supports 8 languages.
* MMS is Romanian-only (per-language Meta checkpoints).
*/

/** All languages that any engine supports. */
Expand All @@ -28,6 +29,7 @@ export const ALL_LANGUAGES = {
no: 'Norwegian',
pl: 'Polish',
pt: 'Portuguese',
ro: 'Romanian',
ru: 'Russian',
sv: 'Swedish',
sw: 'Swahili',
Expand Down Expand Up @@ -70,6 +72,9 @@ export const ENGINE_LANGUAGES: Record<string, readonly LanguageCode[]> = {
tada: ['en', 'ar', 'zh', 'de', 'es', 'fr', 'it', 'ja', 'pl', 'pt'],
kokoro: ['en', 'es', 'fr', 'hi', 'it', 'pt', 'ja', 'zh'],
qwen_custom_voice: ['zh', 'en', 'ja', 'ko', 'de', 'fr', 'ru', 'pt', 'es', 'it'],
mms: ['ro'],
// F5-TTS Romanian community fine-tune — Romanian cloning; retains English.
f5: ['ro', 'en'],
} as const;

/** Helper: get language options for a given engine. */
Expand Down
30 changes: 20 additions & 10 deletions app/src/lib/hooks/useGenerationForm.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ const generationSchema = z.object({
'chatterbox_turbo',
'tada',
'kokoro',
'mms',
'f5',
])
.optional(),
personality: z.boolean().optional(),
Expand Down Expand Up @@ -100,9 +102,13 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
: 'tada-1b'
: engine === 'kokoro'
? 'kokoro'
: engine === 'qwen_custom_voice'
? `qwen-custom-voice-${data.modelSize}`
: `qwen-tts-${data.modelSize}`;
: engine === 'mms'
? 'mms-tts-ron'
: engine === 'f5'
? 'f5-tts-romanian'
: engine === 'qwen_custom_voice'
? `qwen-custom-voice-${data.modelSize}`
: `qwen-tts-${data.modelSize}`;
const displayName =
engine === 'luxtts'
? 'LuxTTS'
Expand All @@ -116,13 +122,17 @@ export function useGenerationForm(options: UseGenerationFormOptions = {}) {
: 'TADA 1B'
: engine === 'kokoro'
? 'Kokoro 82M'
: engine === 'qwen_custom_voice'
? data.modelSize === '1.7B'
? 'Qwen CustomVoice 1.7B'
: 'Qwen CustomVoice 0.6B'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';
: engine === 'mms'
? 'MMS Romanian (Meta)'
: engine === 'f5'
? 'F5-TTS Romanian (community)'
: engine === 'qwen_custom_voice'
? data.modelSize === '1.7B'
? 'Qwen CustomVoice 1.7B'
: 'Qwen CustomVoice 0.6B'
: data.modelSize === '1.7B'
? 'Qwen TTS 1.7B'
: 'Qwen TTS 0.6B';

// Check if model needs downloading
try {
Expand Down
2 changes: 2 additions & 0 deletions app/src/lib/utils/format.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ const ENGINE_DISPLAY_NAMES: Record<string, string> = {
luxtts: 'LuxTTS',
chatterbox: 'Chatterbox',
chatterbox_turbo: 'Chatterbox Turbo',
mms: 'MMS TTS',
f5: 'F5-TTS',
};

export function formatEngineName(engine?: string, modelSize?: string): string {
Expand Down
Loading