notes.md

Notes

• 2026-05-17: Workspace started empty.
• Product direction: agentic language learning app with an always-on tutor, local learner state, autonomous daily lessons/quizzes, answer evaluation, progress updates, and visible 5-10 minute demo loop.
• Implementation choice: start with a small Python CLI loop and deterministic fallback logic so the app runs without external credentials.
• Goal-mode instruction added: keep plan.md current, record real progress and durable decisions here, and adapt calmly if plan.md changes mid-run.
• Built initial Python MVP: local state model, adaptive lesson/quiz engine, auto and interactive CLI modes, bounded demo loop, and a standard-library smoke test.
• Upgraded MVP requirements: data/progress.json is now canonical memory; lessons include vocabulary, grammar, examples; quizzes produce 5-8 mixed question types; console logs show Memory, Lesson Generator, Notification, Adaptive Quiz, Evaluator, Progress Reporter, and Orchestrator agents.
• Added optional fluent_ai.ui Gradio entry point that falls back to console instructions if Gradio is not installed.
• New product direction: add Conversation Mode, a FaceTime-like AI tutor that initiates conversation, adapts topic complexity to measured ability, supports video on/off, and can use camera input such as pointing at an apple to continue the target-language conversation.
• Built Conversation Mode MVP: CLI supports --product-mode conversation, video on/off, visible object labels such as apple, adaptive topic selection, tutor-initiated turns, fluency feedback, and conversation_memory updates in data/progress.json.
• Added .env loading and OpenAI Responses API support. Verified .env is present, OpenAI status reports enabled with gpt-5.5, health check passed, API-backed lesson enhancement worked, and API-backed apple conversation worked.
• Added no-dependency local web app at python -m fluent_ai.web --port 7860; verified browser UI runs Lesson Mode and Conversation Mode with video on and visible object apple.
• Added standalone fullscreen Electron desktop app launched by npm run desktop; it runs outside the browser and calls the existing Python/OpenAI agents.
• 2026-05-17: User asked for clean Duolingo-inspired UI mockups before changing the real app. Added mockups/duolingo-inspired/index.html with three selectable concepts: Learning Path Home, Tutor Call First, and Demo Studio. Added Electron screenshot helper and rendered PNGs for all three concepts. Do not apply a final UI direction until the user chooses one.
• 2026-05-17: Fixed desktop MVP after user reported the app felt non-interactive. Added fluent_ai.desktop_bridge, Electron IPC bridge calls, real lesson quiz submission, turn-by-turn Conversation Mode replies, visible feedback, and logs.
• Verified desktop window through Electron debug context: Conversation Mode accepted "Me llamo Johan.", produced feedback and a follow-up; Lesson Mode accepted quiz answers, graded them, and updated data/progress.json.
• 2026-05-17: User asked to switch the theme to red and apply it. Updated mockups to a red Duolingo-inspired palette, regenerated mockup screenshots, rethemed the richer Electron UI in desktop/electron/renderer.html, rethemed the browser fallback in fluent_ai/web.py, and updated the native Tk Mac app source plus bundled dist/FluentAI.app resource. Verified unit tests, Python compile, localhost status, and browser render.
• 2026-05-17: Fixed Electron window lifetime by keeping a global BrowserWindow reference and rebuilt dist/FluentAI.app as a real Electron app copy instead of the old Tk/Wish wrapper.
• 2026-05-17: Added model selector: default fast local tutor, optional OpenAI model. Live renderer test verified OpenAI conversation and lesson paths use gpt-5.5 from .env.
• 2026-05-17: Added real camera preview request for Video On. Live renderer test showed Camera On with video element and object context apple; conversation used visible object manzana.
• 2026-05-17: Live renderer test results: app starts with zero conversation bubbles, OpenAI tutor accepts learner replies, quiz submits and updates data/progress.json. Screenshots saved in /tmp/fluentai_test_conversation_openai.png and /tmp/fluentai_test_lesson_openai.png.
• 2026-05-17: User asked to confirm the approved design was truly implemented and usable. Found the browser route was still serving the older two-column UI. Fixed it by serving the same red Demo Studio renderer used by Electron and adding /api/bridge/* web endpoints. Updated the live renderer closer to the approved Demo Studio layout: mission/sidebar, large Lesson/Conversation mode cards, main work panel, and dark visible-agent-decision rail. Verified web status, lesson generation, quiz submission, conversation start, video-object fallback, chat reply, no browser console errors, and 7 unit tests. Live verification screenshot saved at mockups/duolingo-inspired/live-app-verified.png.
• 2026-05-17: User asked for a more polished Lesson UI and voice mode by default for Conversation. Updated desktop/electron/renderer.html: Lesson Mode now has a red hero card, metrics, polished vocabulary/example cards, card-style quiz choices, free-answer styling, and improved quiz summary. Conversation Mode now defaults to Voice tutor on, speaks tutor messages with browser speech synthesis, offers a Speak button using SpeechRecognition/webkitSpeechRecognition when available, attempts voice listening after tutor prompts, and keeps typed replies as fallback. Verified tests, browser render, polished lesson DOM, voice default state, conversation bridge start/reply, and saved screenshot mockups/duolingo-inspired/polished-lesson-voice-ui.png.
• 2026-05-17: User clarified Conversation Mode must use an OpenAI voice model, not browser speech synthesis, and video/voice should live in the main chat area like the initial call mockup. Replaced the browser speech hooks with an OpenAI Realtime WebRTC client-secret bridge, added a call-style main stage with Call/Mute/Video/End controls, kept typed chat as Text fallback, and made video context update the main call surface. Video can be selected before a call or toggled into the main stage during an active call. Verified Realtime client-secret creation with gpt-realtime/alloy, browser UI layout, text fallback conversation, no browser console errors, JS syntax checks, and 7 unit tests.
• 2026-05-17: User reported a voice-call error and asked to use the app first. Reproduced the browser-runner failure as microphone permission denial. Fixed Electron media permission handling with both permission-check and permission-request handlers, added a macOS media access IPC bridge, moved mic/camera acquisition before Realtime session creation, and added clear blocked-mic UI copy. Verified the actual Electron app with fake media via remote debugging: Realtime call connected, tutor spoke through gpt-realtime, transcript appeared, video toggled into the main stage, mute changed state, and End returned controls to reconnect.
• 2026-05-17: User rejected the typed visible-object field and asked for OpenAI to process video. Removed the visible-object UI from the Electron/browser renderer, added vision_analyze_frame bridge command, and implemented OpenAI Responses API image input analysis from a captured camera frame. Video calls now analyze a frame, show OpenAI vision context in the call card, and pass that context into the gpt-realtime tutor. Verified in Electron with fake camera/mic: no object input, OpenAI vision saw the fake green camera scene, Realtime tutor used that context in Spanish, video was in the main stage, and the call ended cleanly.
• 2026-05-17: User reported calls failing midway, video not appearing, green shape artifacts, and text fallback not feeling like a real chat. Fixed the Electron renderer so Lesson Mode auto-generates with no setup/top workspace bar, Conversation Mode fills the workspace, OpenAI voice is forced as the startup default, text fallback becomes a full chat interface with a clear return-to-voice control, and video is selected before call startup instead of racing an audio-only call. Removed the green visual accent from the call stage and cleaned stale green shape memory from data/progress.json.
• 2026-05-17: Reworked call startup so the video feed renders immediately, OpenAI Realtime connects first, and OpenAI vision analysis runs only after the voice call is live or when video is toggled into an active call. Verified with the actual Electron app plus fake mic/camera over remote debugging: auto lesson generated, quiz submitted and rendered feedback, Conversation filled the screen, text fallback sent/received messages, return-to-voice worked, video appeared in the main call stage, Realtime reached Live, mute toggled, and End reset controls. Rebuilt dist/FluentAI.app.
• 2026-05-17: User clarified the green shape must not be treated as camera content; video should be the real camera feed and the tutor should respond to what it sees. Added media diagnostics for fake-camera mode, explicit real-camera constraints, live recurring OpenAI vision sampling while video is on, synthetic test-feed filtering so fake green patterns are labeled and not sent as tutor context, and a vision prompt guard against inventing objects from test patterns. Verified Electron with fake media: app labels the feed as Test camera feed, video still renders, Realtime reaches Live, and controls reset. Rebuilt dist/FluentAI.app.
• 2026-05-17: User asked to remove the visible OpenAI enabled with model gpt-5.5 text and make the tutor more responsive when the learner asks for English meaning. Header now shows only level and weak topics; provider status now says OpenAI connected. instead of naming the model. Realtime and text tutor prompts now explicitly answer comprehension/English-help requests in English first, then return to one simple Spanish model phrase. Added a unit test for the English-help fallback and rebuilt dist/FluentAI.app.
• 2026-05-17: Initialized Git, committed the FluentAI project, created public GitHub repo Jo2234/FluentAI, and pushed main. .env, node_modules/, and dist/ are ignored.
• 2026-05-17: Added first-class Hindi, Spanish, and French selection to the desktop/web renderer. The selected language is persisted in local learner state, lessons/quizzes use language-specific content, and text/Realtime/vision prompts now tell the tutor to respond in the language the learner uses when it is Hindi, Spanish, or French, with English explanations for comprehension help.
• 2026-05-17: Made the agent decision log collapsible in the shared desktop/web renderer. The panel stays visible by default for demo transparency, but users can hide it and give the lesson area more room.
• 2026-06-30: Upgraded project hygiene for a stronger public/demo repo: fixed editable packaging with explicit setuptools package discovery, added Python/Node CI, added offline smoke script, added web endpoint smoke test, hardened web JSON parsing/turn bounds, made progress JSON preserve non-ASCII target-language text, and refreshed README quickstart/test/demo docs.
• 2026-06-30: User asked to remove deterministic fallback. Real CLI/web/desktop Lesson and Conversation Mode now require OPENAI_API_KEY; tests and CI use mocked OpenAI providers for reliability without secrets.
• 2026-06-30: User said Realtime tutor cuts them off / talks too soon and should know when to wait, prompt, or explain in English. Added natural turn policy, server VAD silence/idle tuning, no-interrupt setting, language-aware help behavior, stopped camera-context updates from forcing immediate speech, and added regression tests.
• 2026-07-03: Daily product polish finished prior voice/camera work: dynamic Realtime VAD now adapts silence duration by learner level and speaking confidence, camera-frame analysis defaults to fast gpt-4.1-mini with uncertainty guardrails and shorter timeout, video context refreshes more often and shows model/confidence, and the agent decision log starts collapsed to reduce conversation clutter.
• 2026-07-04: Added spaced repetition memory for Lesson Mode. Quiz results now maintain a review_queue in local learner state, due reviews override recent-topic rotation so missed topics come back first, and the desktop/web profile surfaces review count/next review topic. Smoke tests now verify review scheduling.
• 2026-07-05: Daily product polish clarified spaced-review status. Profiles now distinguish reviews due now from the total future schedule, include the next due timestamp, and the desktop/web header says Reviews N due only when immediate practice is needed.
• 2026-07-06: Daily product polish localized Conversation Mode scaffolds for French and Hindi. Beginner topic openings, help model phrases, and known visible-object prompts now use the selected target language instead of Spanish-only fallback text; scripts/smoke_demo.py also runs directly from the README command without needing PYTHONPATH.
• 2026-07-07: Daily product polish localized the remaining Conversation Mode fallback follow-ups and visual auto-replies for French/Hindi, so mocked/offline tutor flows no longer drift back into Spanish after the opening turn.
• 2026-07-08: User requested a substantial UI overhaul after seeing excess empty space, a floating square agent-decision panel, basic lessons, slow quiz feedback, unreadable/generated-looking colors, and Conversation controls overlapping the visual-context card. Updated the shared desktop/web renderer with a full-bleed aurora studio aesthetic, a tall side-rail agent log that pushes content left when expanded, sticky Lesson learning rail, Phrase Lab, per-question answered/correct/review status chips, separated call context/controls, and higher-contrast text/fallback controls.
• 2026-07-08: Synced local main to GitHub origin/main commit 63a3514 (Overhaul FluentAI lesson and call UI), stashed local progress/notes edits as stash@{0}, refreshed Python/Node installs, verified 28 Python tests, smoke demo, and npm run check, then restarted the web app at http://127.0.0.1:7860.
• 2026-07-08: Added docs/FLUENTAI_LIMITLESS_IDEAL.md as the product north-star spec for the ideal unbounded FluentAI: a living, memory-driven, agentic language tutor connecting Lesson Mode, Conversation Mode, voice, video, local learner memory, agent logs, privacy, tests, and roadmap.
• 2026-07-08: Implemented state v2 WP2/WP3 migration: lesson, conversation, bridge, web, CLI, provider, and smoke call sites now use v2 helpers/direct language blocks; compat mirrors were removed from state.py; event IDs now use persistent event_counter; added tests for typed events, v2 review queue, bridge language isolation, bridge delegation, and post-cap event ID uniqueness.
• 2026-07-08: Fixed WP2/WP3 language-switch persistence regression: load_state no longer overrides active language for existing files; bridge language switches persist only when payloads explicitly include language; persisted-file bridge isolation now covers French/no-language/Spanish switching.
• 2026-07-08: Started WP4/WP5 implementation. Constraints: do not touch data/progress.json, preserve existing topic-selection semantics except inserting due mistake memory after due review and before weak topics, keep OpenAI required for real Lesson/Conversation while tests use mocked providers.
• 2026-07-08: Completed WP4/WP5. Lessons now carry reason from due review, due mistake memory, weak topic, or rotation; desktop/web/CLI surfaces show the reason. Quiz results now include deterministic error categories, corrected forms, severity, and confidence; desktop lesson submit can use mocked/OpenAI grading for fill-blank/open-ended answers with local fallback. Lesson misses call record_mistake, schedule next_review, and can drive the next lesson before weak-topic selection.
• 2026-07-08: Completed WP6/WP7. Lesson misses now set conversation_memory.next_conversation_goal and clean lessons clear stale lesson-sourced goals. Conversation topic selection consumes goals unless video context wins; tutor prompts and realtime instructions include goal guidance. Conversation corrections now store real learner text, map conversation topics to teachable lesson-bank topics (likes and food/weather/visible object -> vocabulary, past weekend -> past tense, direct lesson topics unchanged), and schedule review. Text fallback, voice conversation_end, web, and CLI now persist capped post-call summaries.
• 2026-07-08: Implemented WP4 Home workspace and privacy controls: home_summary, sanitized memory inspect/export, language reset, delete-all memory, Electron save-dialog export, and home-first renderer flow.
• 2026-07-08: Packaging steps 2/3 in progress: added Electron API key gate/IPC, safeStorage or session-only key handling, bridge validate_key, secret redaction, and corrupt progress.json backup/recovery notice path.
• 2026-07-08: Completed Packaging step 6 reliability pass: bridge session checkpoints, empty tutor retry/recovery, model-failure copy, renderer fetch timeouts/retry policy, realtime refresh scaffolding/fallback, camera voice-only fallback, tests, and live interrupted-lesson resume verification.
• 2026-07-08: Consumer acceptance run attempted against packaged dist/mac-arm64/FluentAI.app at cf9c4d5 with isolated FLUENTAI_USER_DATA_PATH and FLUENTAI_FAKE_MEDIA=1. Host GUI capture/control was blocked (computer-use timeout/AX cannotComplete, screencapture could not create image, System Events hung, packaged Electron CDP endpoint hung), so true screenshot-based GUI acceptance is not complete. Packaged bridge fallback verified onboarding, placement, home summary, lesson feedback/adaptation, checkpoint resume, and schema-v2/no-secret state in temp profile. Defects found: Realtime client secret failed in packaged bridge with Python SSL certificate verification error; text tutor visibly corrected yo estudiar por la manana but structured turn metadata/mistake memory did not record that conversation mistake.
• 2026-07-08: Fixed the two acceptance-run defects. Direct provider HTTPS calls now use a certifi SSL context inside the PyInstaller bridge. Conversation turns can use one strict provider JSON grade per learner reply, falling back to the local keyword scorer on malformed output; provider correction forms/categories now flow through existing mistake memory and post-call summary paths. Verified python3 -m pytest (111 passed), npm run check, ./scripts/build_bridge.sh, packaged realtime_client_secret with real .env key redacted (ok, model gpt-realtime, secret present), and packaged vision_analyze_frame no-key clean error.
• 2026-07-08: Fixed conversation-to-lesson mistake scheduling. Conversation corrections now make mistake-memory reviews immediately due; lesson-origin misses still schedule next review for +1 day. Successful mistake-memory lessons push the practiced mistake topic out by +1 day so the same due mistake does not loop forever.
• 2026-07-08: Final polish added phrase audio playback through cached OpenAI TTS, lesson pronunciation/culture cards, OpenAI enhancement context preservation for those fields, and post-call pronunciation practice notes; verified npm run check and one real phrase_audio call (ok, 36096 bytes).
• 2026-07-08: Fixed Lesson Mode quiz contrast in the aurora renderer. Multiple-choice labels now render light text on dark cards with subtle hover and indigo selected states; prompts, placeholders, feedback, and lesson Try It/listen controls were swept for dark-on-dark. Verified python -m unittest discover -s tests -q, npm run test:js, and a temp-state live web drive measuring a generated choice label at rgb(238, 242, 255) on rgba(15, 16, 22, 0.78).
• 2026-07-08: Daily product polish made practice streaks real instead of static profile copy. Lesson and Conversation Mode now update language-specific streak_days, last_practice_date, last_session_at, and reset same-language daily counters when a new UTC practice day starts.