🎙️ VoxRoute — Local Real-Time Travel Voice AI

A fully local, runnable MVP of a real-time travel-support voice AI agent. VoxRoute demonstrates the core engineering challenges of a production voice support system — all running on your laptop without paid APIs or cloud inference.

⚠️ DEMO SYSTEM — All travel information is sample data for demonstration purposes.
   Not official government or agency information.

---

🏗️ System Architecture

flowchart TB
    subgraph Frontend["🖥️ Streamlit Frontend"]
        UI_Mic["🎤 Audio Input / Mic Recorder"]
        UI_Transcript["💬 Live Transcript View"]
        UI_State["🧠 State & Metrics Dashboard"]
        UI_Player["🔊 Audio Playback"]
    end

    subgraph AgentOrchestrator["⚡ Agent Orchestrator (Async)"]
        direction TB
        VAD["🛡️ Silero VAD<br/>(Voice Activity Detection)"]
        STT["🎙️ faster-whisper<br/>(Local STT Engine)"]
        Intent["🎯 Intent Detector<br/>(Ollama / Regex Fallback)"]
        Memory["🧠 Conversation Memory<br/>(Entity Resolution & History)"]
        StateMachine["🔄 Agent State Machine<br/>(Transitions & Error Recovery)"]
        Retriever["📚 RAG Retriever<br/>(Query Optimizer)"]
        ToolRouter["⚙️ Tool Execution Router<br/>(Mock Visa DB & API)"]
        LLM["🤖 Ollama Engine<br/>(qwen2.5:3b)"]
        Guard["🛡️ Response Guard<br/>(Evidence & Grounding Check)"]
        TTS["🗣️ Piper / pyttsx3<br/>(Local TTS Synthesis)"]
        Interrupt["⏹️ Interruption Handler<br/>(Half-Duplex Playback Control)"]
    end

    subgraph Knowledge["📂 Local Knowledge & Data"]
        Chroma[("🗄️ ChromaDB Vector Store<br/>(all-MiniLM-L6-v2)")]
        DocFiles["📄 Demo Travel FAQs<br/>& Visa Policy Docs"]
    end

    subgraph Persistence["💾 Persistence Layer"]
        SQLite[("🗃️ SQLite Database<br/>(Conversations, Metrics, Failures)")]
    end

    UI_Mic -->|WAV Audio Bytes| VAD
    VAD --> STT
    STT -->|Transcript| Intent
    Intent <--> Memory
    Intent --> StateMachine
    Intent --> Retriever
    Retriever <--> Chroma
    DocFiles -->|Ingestion| Chroma
    Intent --> ToolRouter
    Retriever -->|Grounded Context| LLM
    ToolRouter -->|Tool Output JSON| LLM
    Memory -->|Context Window| LLM
    LLM -->|Draft Response| Guard
    Guard -->|Validated Text| TTS
    TTS -->|WAV Buffer| UI_Player
    UI_Player -.->|Stop / Interrupt| Interrupt
    Interrupt -.->|Cancel Playback| TTS

    AgentOrchestrator -->|Log Turns & Performance| SQLite
    SQLite -->|Aggregate Stats & Metrics| UI_State
    AgentOrchestrator -->|Live Updates| UI_Transcript

Loading

---

🔄 End-to-End Voice Pipeline

flowchart LR
    Mic(["🎙️ Microphone"]) --> VAD["Silero VAD\nFilter Silence"]
    VAD --> STT["faster-whisper\nSpeech to Text"]
    STT --> Intent{"Intent & Entities\n(Ollama LLM)"}
    
    Intent -->|Query Context| RAG["ChromaDB\nKnowledge Retrieval"]
    Intent -->|Requires Tool| Tools["Local Travel Tools\n(Status, Docs, Time)"]
    
    RAG --> Gen["Ollama LLM\nResponse Generation"]
    Tools --> Gen
    
    Gen --> Guard{"Response Guard\nGrounded & Safe?"}
    Guard -->|Pass / Adjusted| TTS["Piper TTS\nSpeech Synthesis"]
    Guard -->|Low Confidence / Risk| Escalation["Escalation Router\n(Human Agent Handoff)"]
    
    TTS --> Speaker(["🔊 Audio Speaker"])
    Escalation --> TTS

Loading

---

🔄 Agent State Machine

stateDiagram-v2
    [*] --> IDLE
    IDLE --> LISTENING : Start Conversation
    LISTENING --> TRANSCRIBING : User Speaks (Audio Input)
    TRANSCRIBING --> UNDERSTANDING : Transcript Available
    TRANSCRIBING --> ERROR_RECOVERY : STT Failed
    
    UNDERSTANDING --> RETRIEVING : Need Knowledge Base
    UNDERSTANDING --> TOOL_CALL : Tool Triggered
    UNDERSTANDING --> GENERATING : Direct Query
    UNDERSTANDING --> ESCALATION : Explicit Human Request
    
    RETRIEVING --> TOOL_CALL : Tool Also Needed
    RETRIEVING --> GENERATING : Context Ready
    
    TOOL_CALL --> GENERATING : Tool Output Ready
    TOOL_CALL --> ESCALATION : Tool Failed
    
    GENERATING --> VALIDATING : Draft Answer Ready
    
    VALIDATING --> SPEAKING : Validation Passed
    VALIDATING --> GENERATING : Unsafe Certainty Adjusted
    VALIDATING --> ESCALATION : Low Evidence / Hallucination
    
    SPEAKING --> LISTENING : Playback Completed / Interrupted
    SPEAKING --> IDLE : End Conversation
    
    ESCALATION --> SPEAKING : Synthesize Handoff Notice
    ERROR_RECOVERY --> LISTENING : Fallback Recovery

Loading

---

⏹️ Interruption Handling Flow

sequenceDiagram
    autonumber
    actor Customer as 🧑 Customer
    participant UI as 🖥️ Streamlit UI
    participant Handler as ⏹️ Interruption Handler
    participant TTS as 🗣️ TTS / Audio Engine
    participant Agent as ⚡ Orchestrator

    Agent->>TTS: Synthesize and begin playback
    TTS-->>UI: Streaming audio / playback active
    Note over Customer,TTS: Agent is speaking...
    
    Customer->>UI: Clicks "Stop" or Records new utterance
    UI->>Handler: interrupt(reason="user_button")
    Handler->>TTS: stop_playback() (Set Stop Event)
    TTS-->>UI: Playback terminated immediately
    Handler->>Agent: Transition state: SPEAKING → LISTENING
    UI->>Agent: Submit new customer utterance
    Agent->>Agent: Process new turn

Loading

---

🛠️ Technology Stack

Component	Technology	Purpose
Frontend	Streamlit	Web UI with audio I/O
STT	faster-whisper	Local speech-to-text
LLM	Ollama (qwen2.5:3b)	Local language model
Embeddings	sentence-transformers	Document embeddings
Vector DB	ChromaDB	RAG retrieval
TTS	Piper TTS / pyttsx3	Local text-to-speech
VAD	Silero VAD	Voice activity detection
Database	SQLite	Conversation & metrics logging

---

📦 Installation

Prerequisites

• Python 3.10+
• Ollama — https://ollama.ai
• Microphone (for voice mode)

Step 1: Clone and Setup

git clone <repository>
cd voxroute

# Create virtual environment
python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Step 2: Install and Start Ollama

1. Download and install Ollama from https://ollama.ai
2. Start the Ollama service:

ollama serve

3. Pull the default model:

ollama pull qwen2.5:3b

Step 3: Configure (Optional)

# Copy the example config
copy .env.example .env    # Windows
# cp .env.example .env    # Linux/macOS

# Edit .env to customize models, thresholds, etc.

Step 4: Piper TTS (Optional but Recommended)

Piper provides better quality local TTS. If it fails to install via pip, the app will fall back to system TTS (pyttsx3).

pip install piper-tts

If pip installation fails on Windows, you can:

1. Download the Piper binary from Piper releases
2. Download a voice model from Piper voices
3. Place the .onnx and .onnx.json files in models/piper/

The app will automatically fall back to pyttsx3 (system TTS) if Piper is not available.

Step 5: Run

python run.py

This single command:

1. Checks Python version
2. Verifies Ollama is running
3. Creates .env if missing
4. Launches the Streamlit app

The app opens at http://localhost:8501.

---

🎮 Usage

Voice Mode

1. Click Start Conversation
2. Click the microphone widget to record your message
3. The system processes through the full pipeline:
   • STT → Intent → RAG → Tool → LLM → Guard → TTS
4. You'll hear the response and see real-time metrics

Text Mode (Demo)

1. Switch to the Text Mode tab
2. Use the quick demo buttons or type your own query
3. The system processes through the pipeline (skipping STT)

Evaluation Dashboard

• Switch to the Evaluation tab
• See aggregate metrics, latency breakdowns, and failure patterns

---

🧪 Demo Scenarios

Scenario 1 — Normal Query

You: "How long does a France visa normally take?"

Expected: Intent → VISA_PROCESSING_TIME, RAG retrieval, grounded response about 7-15 working days

Scenario 2 — Tool Calling

You: "Can you check application AT12345?"

Expected: Intent → VISA_STATUS, Tool → check_application_status, status from mock database

Scenario 3 — Hallucination Prevention

You: "Will my visa definitely arrive tomorrow?"

Expected: Response Guard catches absolute certainty, modifies response to use hedged language

Scenario 4 — Human Escalation

You: "I want to speak to a human."

Expected: Intent → HUMAN_REQUEST, handoff_to_human() tool called, escalation message

Scenario 5 — Interruption

During agent playback, click End Conversation or record a new message

Expected: TTS stops, new input is processed

---

📁 Project Structure

voxroute/
├── app.py                      # Streamlit main application
├── run.py                      # Single entry point
├── config.py                   # Central configuration
├── requirements.txt
├── .env.example
├── README.md
│
├── agent/
│   ├── orchestrator.py         # Main pipeline coordinator
│   ├── intent.py               # LLM-based intent detection
│   ├── state.py                # Agent state machine
│   ├── memory.py               # Conversation memory
│   └── prompts.py              # All LLM prompts
│
├── voice/
│   ├── stt.py                  # faster-whisper STT
│   ├── tts.py                  # Piper/pyttsx3 TTS
│   ├── vad.py                  # Silero VAD
│   └── interruption.py         # Interruption handler
│
├── rag/
│   ├── vectorstore.py          # ChromaDB management
│   ├── ingest.py               # Document chunking & indexing
│   └── retriever.py            # RAG retrieval
│
├── tools/
│   ├── visa.py                 # Processing times & status
│   ├── documents.py            # Required documents
│   ├── eligibility.py          # Visa eligibility
│   └── support.py              # Tickets & human handoff
│
├── safety/
│   ├── response_guard.py       # Response validation
│   └── confidence.py           # Escalation logic
│
├── evaluation/
│   ├── metrics.py              # Quality metrics
│   └── failure_analyzer.py     # Failure classification
│
├── database/
│   └── db.py                   # SQLite manager
│
├── dashboard/
│   └── components.py           # Streamlit UI components
│
├── backend/
│   ├── server.py               # Startup & health checks
│   └── websocket.py            # WebSocket (future)
│
└── data/
    ├── travel_faq.txt           # Demo FAQ data
    ├── visa_processing.txt      # Demo processing times
    ├── required_documents.txt   # Demo document requirements
    └── travel_support.txt       # Demo support info

---

⚙️ Configuration

All settings are configurable via .env:

Variable	Default	Description
WHISPER_MODEL	small	STT model (tiny/base/small/medium/large-v3)
OLLAMA_MODEL	qwen2.5:3b	Ollama LLM model
OLLAMA_BASE_URL	http://localhost:11434	Ollama API URL
EMBEDDING_MODEL	all-MiniLM-L6-v2	Embedding model
PIPER_VOICE	en_US-lessac-medium	Piper voice model
TTS_FALLBACK	true	Fall back to pyttsx3 if Piper unavailable
RAG_TOP_K	3	Number of RAG chunks to retrieve
MAX_CONVERSATION_TURNS	10	Memory window size
CONFIDENCE_HIGH	0.85	Threshold for direct answer
CONFIDENCE_MEDIUM	0.60	Threshold for clarification
FORCE_CPU	false	Force CPU even with CUDA available

Hardware Detection

The app automatically detects CUDA availability:

• CUDA GPU: Uses float16 compute for STT, GPU embeddings
• CPU only: Uses int8 compute for STT, CPU embeddings

---

🔧 Troubleshooting

"Cannot connect to Ollama"

# Start Ollama
ollama serve

# Verify it's running
curl http://localhost:11434/api/tags

"Model not found"

# Pull the configured model
ollama pull qwen2.5:3b

"faster-whisper not found"

pip install faster-whisper

"No audio output"

• Install sounddevice: pip install sounddevice
• Check system audio output settings
• In text mode, audio plays via Streamlit's st.audio widget

"Piper TTS not working on Windows"

• The piper-tts pip package may not have Windows wheels
• The app automatically falls back to pyttsx3 (system TTS)
• For better quality, download the Piper binary from GitHub releases

High Latency

• Use a smaller Whisper model: WHISPER_MODEL=tiny
• Use a smaller LLM: OLLAMA_MODEL=qwen2.5:1.5b
• If CUDA is available, it will be used automatically

ChromaDB Errors

# Delete the vector store and re-ingest
rm -rf data/chroma_db/
python run.py

---

📊 Evaluation Methodology

Per-Turn Metrics

• Intent Accuracy: Based on LLM confidence score
• Response Groundedness: Keyword overlap between response and evidence
• Tool Success: Whether tool calls return successful results
• Latency: Pipeline stage timing (STT + Intent + RAG + Tool + LLM + Guard + TTS)

Aggregate Metrics

• Total conversations, average latencies
• Tool success rate, escalation rate
• Grounded response rate, resolution rate

Failure Classification

Failures are automatically categorized:

• STT_FAILURE: Low confidence or empty transcription
• INTENT_FAILURE: Unknown intent or low confidence
• RAG_FAILURE: No relevant chunks found
• TOOL_FAILURE: Tool returned error
• HALLUCINATION: Response failed grounding check
• HIGH_LATENCY: > 5 second response time
• HUMAN_ESCALATION: Conversation escalated

---

⚠️ Known Limitations

1. Half-duplex interruption: User must click stop/re-record rather than auto-detecting voice during playback (true full-duplex requires WebRTC)
2. Piper TTS Windows: May need manual binary installation; pyttsx3 fallback is available
3. LLM tool calling: Uses deterministic intent→tool mapping rather than LLM native function calling (more reliable with small models)
4. CPU latency: On CPU-only machines, expect 3-10 second total response time
5. Single-turn audio: Each recording is processed as a complete utterance (no streaming)
6. Demo data only: All travel information is fictional sample data

---

🔒 Privacy

VoxRoute runs entirely locally:

• No data sent to cloud services
• No API keys required
• All models run on your machine
• Conversations stored only in local SQLite
• Internet only needed for initial model downloads

---

📝 License

MIT License — see LICENSE file for details.