Docs: Update CLAUDE.md with recent improvements and enforce uv run usage
Browse files## Changes
**CLAUDE.md updates:**
- Add "Recent Improvements" section documenting 6 major changes from Dec 2025 - Jan 2026
- Update all development workflow commands to use `uv run` prefix
- Fix path references: `src/reachy_mini_language_tutor/` → `reachy_mini_language_tutor/`
- Add privacy protections documentation for persistent memory
- Update tool documentation to reflect recall/remember moved to shared tools/
- Emphasize `uv run` for ruff, mypy, pytest in Code Style Conventions
**GEMINI.md:**
- Replace duplicate file with symlink to CLAUDE.md for consistency
Improvements documented:
- Privacy protections for persistent memory (Jan 2026)
- Normalized language tutor profiles with English-first approach (Jan 2026)
- Shared memory tools eliminating 455 lines of duplication (Jan 2026)
- Source code migration to root directory (Dec 2025)
- SDK compatibility checks (Jan 2026)
- Template-based tutor design with 64-85% size reduction (Dec 2025)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
@@ -15,6 +15,20 @@ Reachy Language Partner - a language learning companion for the Reachy Mini robo
|
|
| 15 |
|
| 16 |
Powered by OpenAI's realtime API, vision processing, and choreographed motion.
|
| 17 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
## Important Resources
|
| 19 |
|
| 20 |
**ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
|
|
@@ -58,12 +72,14 @@ reachy-mini-language-tutor --profile <name> # Load custom profile
|
|
| 58 |
```
|
| 59 |
|
| 60 |
### Development Workflow
|
|
|
|
|
|
|
| 61 |
```bash
|
| 62 |
-
ruff check . # Lint and format check
|
| 63 |
-
ruff format . # Auto-format code
|
| 64 |
-
mypy
|
| 65 |
-
pytest # Run test suite
|
| 66 |
-
pytest tests/test_openai_realtime.py # Run specific test file
|
| 67 |
```
|
| 68 |
|
| 69 |
## Architecture
|
|
@@ -136,7 +152,7 @@ Six language tutor profiles available in `profiles/`:
|
|
| 136 |
- **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
|
| 137 |
- **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
|
| 138 |
|
| 139 |
-
Each profile in `
|
| 140 |
- **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
|
| 141 |
- **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
|
| 142 |
- **`proactive.txt`**: Set to `true` for proactive greeting mode
|
|
@@ -231,8 +247,15 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
|
|
| 231 |
|
| 232 |
**Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
|
| 233 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 234 |
## Available LLM Tools
|
| 235 |
|
|
|
|
|
|
|
| 236 |
| Tool | Action | Dependencies |
|
| 237 |
|------|--------|--------------|
|
| 238 |
| `move_head` | Queue head pose change (left/right/up/down/front) | Core |
|
|
@@ -248,21 +271,22 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
|
|
| 248 |
|
| 249 |
## Code Style Conventions
|
| 250 |
|
| 251 |
-
- **Type checking**: Strict mypy enabled (`mypy
|
| 252 |
-
- **Formatting**: Ruff with 119-char line length
|
| 253 |
- **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
|
| 254 |
- **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
|
| 255 |
- **Quote style**: Double quotes
|
| 256 |
- **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
|
|
|
|
| 257 |
|
| 258 |
## Development Tips
|
| 259 |
|
| 260 |
### Adding Custom Tools
|
| 261 |
-
1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`)
|
| 262 |
2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
|
| 263 |
3. Implement `name`, `description`, `parameters`, and `__call__()` method
|
| 264 |
4. Add tool name to `profiles/<profile_name>/tools.txt`
|
| 265 |
-
5. See `
|
| 266 |
|
| 267 |
### Motion Control Principles
|
| 268 |
- **100Hz loop is sacred** - never block the `MovementManager` thread
|
|
@@ -331,7 +355,8 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
|
|
| 331 |
- Tests located in `tests/`
|
| 332 |
- Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
|
| 333 |
- Audio fixtures available in `conftest.py`
|
| 334 |
-
- Run with `pytest` after `uv sync --group dev`
|
|
|
|
| 335 |
|
| 336 |
## Dependencies & Extras
|
| 337 |
|
|
|
|
| 15 |
|
| 16 |
Powered by OpenAI's realtime API, vision processing, and choreographed motion.
|
| 17 |
|
| 18 |
+
## Recent Improvements
|
| 19 |
+
|
| 20 |
+
**Privacy Protections (Jan 2026)**: Added comprehensive privacy guidelines to persistent memory system. Memory now implements moderate privacy level with clear allowed/excluded data categories.
|
| 21 |
+
|
| 22 |
+
**Normalized Language Tutor Profiles (Jan 2026)**: All language tutors now follow consistent English-first instruction approach. Cultural content is properly framed within the teaching methodology rather than being a primary focus.
|
| 23 |
+
|
| 24 |
+
**Shared Memory Tools (Jan 2026)**: Moved `recall` and `remember` tools from profile-specific implementations to shared `tools/` directory, eliminating 455 lines of duplicate code and fixing "unknown tool" errors for default profile.
|
| 25 |
+
|
| 26 |
+
**Source Code Migration (Dec 2025)**: Migrated source code from `src/reachy_mini_language_tutor/` to root-level `reachy_mini_language_tutor/` directory for cleaner project structure.
|
| 27 |
+
|
| 28 |
+
**SDK Compatibility Check (Jan 2026)**: Added defensive version checking for SDK methods (e.g., `clear_output_buffer()`), preventing crashes when using different SDK versions.
|
| 29 |
+
|
| 30 |
+
**Template-Based Tutor Design (Dec 2025)**: Extracted 13 shared prompt components (~165 lines) into reusable templates, reducing tutor file sizes by 64-85% and ensuring consistent methodology across all language tutors.
|
| 31 |
+
|
| 32 |
## Important Resources
|
| 33 |
|
| 34 |
**ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
|
|
|
|
| 72 |
```
|
| 73 |
|
| 74 |
### Development Workflow
|
| 75 |
+
**IMPORTANT: Always use `uv run` to execute development tools to ensure they run in the correct virtual environment.**
|
| 76 |
+
|
| 77 |
```bash
|
| 78 |
+
uv run ruff check . # Lint and format check
|
| 79 |
+
uv run ruff format . # Auto-format code
|
| 80 |
+
uv run mypy reachy_mini_language_tutor/ # Type checking (strict mode enabled)
|
| 81 |
+
uv run pytest # Run test suite
|
| 82 |
+
uv run pytest tests/test_openai_realtime.py # Run specific test file
|
| 83 |
```
|
| 84 |
|
| 85 |
## Architecture
|
|
|
|
| 152 |
- **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
|
| 153 |
- **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
|
| 154 |
|
| 155 |
+
Each profile in `reachy_mini_language_tutor/profiles/<name>/` contains:
|
| 156 |
- **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
|
| 157 |
- **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
|
| 158 |
- **`proactive.txt`**: Set to `true` for proactive greeting mode
|
|
|
|
| 247 |
|
| 248 |
**Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
|
| 249 |
|
| 250 |
+
**Privacy Protections**: The memory system implements moderate privacy guidelines:
|
| 251 |
+
- **ALLOWED**: First names, general region, occupation category, interests, learning data
|
| 252 |
+
- **EXCLUDED**: Age, specific location, family names, sensitive details, travel dates
|
| 253 |
+
- Implicit consent model via `SUPERMEMORY_API_KEY` configuration
|
| 254 |
+
|
| 255 |
## Available LLM Tools
|
| 256 |
|
| 257 |
+
**Note**: The `recall` and `remember` tools are now in the shared `tools/` directory (as of recent refactor), making them available to all profiles without duplication.
|
| 258 |
+
|
| 259 |
| Tool | Action | Dependencies |
|
| 260 |
|------|--------|--------------|
|
| 261 |
| `move_head` | Queue head pose change (left/right/up/down/front) | Core |
|
|
|
|
| 271 |
|
| 272 |
## Code Style Conventions
|
| 273 |
|
| 274 |
+
- **Type checking**: Strict mypy enabled (`uv run mypy reachy_mini_language_tutor/`)
|
| 275 |
+
- **Formatting**: Ruff with 119-char line length (`uv run ruff format .`)
|
| 276 |
- **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
|
| 277 |
- **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
|
| 278 |
- **Quote style**: Double quotes
|
| 279 |
- **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
|
| 280 |
+
- **Tool execution**: Always use `uv run` for ruff, mypy, pytest, and other dev tools
|
| 281 |
|
| 282 |
## Development Tips
|
| 283 |
|
| 284 |
### Adding Custom Tools
|
| 285 |
+
1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`) for profile-specific tools, or in `tools/` for shared tools
|
| 286 |
2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
|
| 287 |
3. Implement `name`, `description`, `parameters`, and `__call__()` method
|
| 288 |
4. Add tool name to `profiles/<profile_name>/tools.txt`
|
| 289 |
+
5. See `tools/recall.py` and `tools/remember.py` for examples of shared tools available to all profiles
|
| 290 |
|
| 291 |
### Motion Control Principles
|
| 292 |
- **100Hz loop is sacred** - never block the `MovementManager` thread
|
|
|
|
| 355 |
- Tests located in `tests/`
|
| 356 |
- Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
|
| 357 |
- Audio fixtures available in `conftest.py`
|
| 358 |
+
- Run with `uv run pytest` after `uv sync --group dev`
|
| 359 |
+
- Always use `uv run pytest` to ensure tests run in the correct environment
|
| 360 |
|
| 361 |
## Dependencies & Extras
|
| 362 |
|
|
@@ -1,356 +0,0 @@
|
|
| 1 |
-
# GEMINI.md
|
| 2 |
-
|
| 3 |
-
This file provides guidance to Gemini Code (google.ai/gemini/code) when working with code in this repository.
|
| 4 |
-
|
| 5 |
-
## Project Overview
|
| 6 |
-
Reachy Language Partner - a language learning companion for the Reachy Mini robot. Practice conversational skills in French, Spanish, German, Italian, Portuguese, and other languages through natural dialogue with an expressive robot partner.
|
| 7 |
-
|
| 8 |
-
**Key Features:**
|
| 9 |
-
- Persistent memory across sessions (tracks progress, struggles, preferences)
|
| 10 |
-
- Proactive engagement and gentle correction through recasting
|
| 11 |
-
- Grammar deep-dive mode with complete explanations on demand
|
| 12 |
-
- Error pattern tracking with proactive review at session start
|
| 13 |
-
- Session summaries with highlights and next-steps recommendations
|
| 14 |
-
- Expressive robot feedback (dances, emotions, celebrations)
|
| 15 |
-
|
| 16 |
-
Powered by OpenAI's realtime API, vision processing, and choreographed motion.
|
| 17 |
-
|
| 18 |
-
## Important Resources
|
| 19 |
-
|
| 20 |
-
**ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
|
| 21 |
-
- **SDK Documentation**: https://github.com/pollen-robotics/reachy_mini/blob/develop/docs/SDK/readme.md
|
| 22 |
-
|
| 23 |
-
When working on robot control, motion systems, hardware interfaces, or any Reachy Mini-specific functionality, refer to the official SDK documentation first. This is the authoritative source for:
|
| 24 |
-
- Robot API reference and methods
|
| 25 |
-
- Hardware capabilities and limitations
|
| 26 |
-
- Control loop best practices
|
| 27 |
-
- Joint limits and coordinate systems
|
| 28 |
-
- Official code examples and patterns
|
| 29 |
-
|
| 30 |
-
## Build & Run Commands
|
| 31 |
-
|
| 32 |
-
### Installation
|
| 33 |
-
```bash
|
| 34 |
-
# Using uv (recommended)
|
| 35 |
-
uv venv --python 3.12.1
|
| 36 |
-
source .venv/bin/activate
|
| 37 |
-
uv sync # Base install
|
| 38 |
-
uv sync --extra all_vision # With all vision features
|
| 39 |
-
uv sync --extra reachy_mini_wireless # For wireless robot support
|
| 40 |
-
uv sync --group dev # Add dev tools (pytest, ruff, mypy)
|
| 41 |
-
|
| 42 |
-
# Using pip
|
| 43 |
-
python -m venv .venv
|
| 44 |
-
source .venv/bin/activate
|
| 45 |
-
pip install -e .
|
| 46 |
-
pip install -e .[all_vision,dev]
|
| 47 |
-
```
|
| 48 |
-
|
| 49 |
-
### Running the App
|
| 50 |
-
```bash
|
| 51 |
-
reachy-mini-language-tutor # Console mode (default)
|
| 52 |
-
reachy-mini-language-tutor --gradio # Web UI at http://127.0.0.1:7860/
|
| 53 |
-
reachy-mini-language-tutor --head-tracker mediapipe # With face tracking
|
| 54 |
-
reachy-mini-language-tutor --local-vision # Local SmolVLM2 vision
|
| 55 |
-
reachy-mini-language-tutor --wireless-version # For wireless robot
|
| 56 |
-
reachy-mini-language-tutor --no-camera # Audio-only mode
|
| 57 |
-
reachy-mini-language-tutor --profile <name> # Load custom profile
|
| 58 |
-
```
|
| 59 |
-
|
| 60 |
-
### Development Workflow
|
| 61 |
-
```bash
|
| 62 |
-
ruff check . # Lint and format check
|
| 63 |
-
ruff format . # Auto-format code
|
| 64 |
-
mypy reachy_mini_language_tutor/ # Type checking (strict mode enabled)
|
| 65 |
-
pytest # Run test suite
|
| 66 |
-
pytest tests/test_openai_realtime.py # Run specific test file
|
| 67 |
-
```
|
| 68 |
-
|
| 69 |
-
## Architecture
|
| 70 |
-
|
| 71 |
-
### Layered Motion System (Core Design)
|
| 72 |
-
The robot uses a **compose-based motion blending** system with primary and secondary layers:
|
| 73 |
-
|
| 74 |
-
**Primary Moves** (sequential, mutually exclusive queue in `MovementManager`):
|
| 75 |
-
- Dances (from `reachy_mini_dances_library`)
|
| 76 |
-
- Recorded emotions (Hugging Face assets)
|
| 77 |
-
- Goto poses (head positioning)
|
| 78 |
-
- Breathing (idle state fallback)
|
| 79 |
-
|
| 80 |
-
**Secondary Offsets** (additive, blended in real-time):
|
| 81 |
-
- Speech wobble (`HeadWobbler`) - reactive to audio stream activity
|
| 82 |
-
- Face tracking offsets (`CameraWorker`) - smooth interpolation with 2s fade-out when face is lost
|
| 83 |
-
|
| 84 |
-
**Control Loop**: `MovementManager` runs at **100Hz** in a dedicated thread, composing primary + secondary poses and calling `robot.set_target()`.
|
| 85 |
-
|
| 86 |
-
### Threading Model
|
| 87 |
-
- **Main thread**: Launches Gradio/console UI
|
| 88 |
-
- **MovementManager thread**: 100Hz motion control loop
|
| 89 |
-
- **HeadWobbler thread**: Audio-reactive motion processing (speech detection)
|
| 90 |
-
- **CameraWorker thread**: 30Hz+ frame capture + face tracking
|
| 91 |
-
- **VisionManager thread**: Periodic local VLM inference (if `--local-vision` enabled)
|
| 92 |
-
|
| 93 |
-
### Tool Dispatch Pattern
|
| 94 |
-
```
|
| 95 |
-
User audio → OpenaiRealtimeHandler (24kHz mono WebRTC)
|
| 96 |
-
→ LLM generates tool calls
|
| 97 |
-
→ dispatch_tool_call() routes to Tool subclass
|
| 98 |
-
→ Tool enqueues command in MovementManager
|
| 99 |
-
→ MovementManager executes and returns status via AdditionalOutputs
|
| 100 |
-
→ Response audio + motion sent to robot/user
|
| 101 |
-
```
|
| 102 |
-
|
| 103 |
-
### Vision Pipeline Options
|
| 104 |
-
- **Default (gpt-realtime)**: Camera tool sends frames to OpenAI for vision
|
| 105 |
-
- **Local vision (`--local-vision`)**: VisionManager processes frames with SmolVLM2 (on-device CPU/GPU/MPS)
|
| 106 |
-
- **Face tracking options**:
|
| 107 |
-
- `--head-tracker yolo`: YOLOv8-based face detection (requires `yolo_vision` extra)
|
| 108 |
-
- `--head-tracker mediapipe`: MediaPipe from `reachy_mini_toolbox` (requires `mediapipe_vision` extra, pinned to 0.10.14)
|
| 109 |
-
|
| 110 |
-
## Key File Responsibilities
|
| 111 |
-
|
| 112 |
-
| File/Directory | Purpose |
|
| 113 |
-
|----------------|---------|
|
| 114 |
-
| `main.py` | Entry point - initializes robot, managers, UI (Gradio or console) |
|
| 115 |
-
| `openai_realtime.py` | `OpenaiRealtimeHandler` - async WebRTC stream to OpenAI realtime API |
|
| 116 |
-
| `moves.py` | `MovementManager` - 100Hz motion control loop with queue management |
|
| 117 |
-
| `camera_worker.py` | `CameraWorker` - thread-safe frame buffering + face tracking offset calculation |
|
| 118 |
-
| `audio/head_wobbler.py` | Speech-reactive head wobble (additive motion from audio stream) |
|
| 119 |
-
| `tools/core_tools.py` | `Tool` base class + dispatch logic for LLM tool calls |
|
| 120 |
-
| `tools/` | Built-in tools: `move_head`, `dance`, `play_emotion`, `camera`, `head_tracking`, etc. |
|
| 121 |
-
| `config.py` | `Config` class - loads `.env` and manages profile settings |
|
| 122 |
-
| `prompts.py` | Prompt loading with `[placeholder]` syntax for reusable components |
|
| 123 |
-
| `console.py` | `LocalStream` - headless mode with direct audio I/O + settings routes |
|
| 124 |
-
| `gradio_personality.py` | `PersonalityUI` - profile selector, creator, live instruction reload |
|
| 125 |
-
| `vision/processors.py` | Local vision manager (SmolVLM2 initialization) |
|
| 126 |
-
| `vision/yolo_head_tracker.py` | YOLO-based face detection for head tracking |
|
| 127 |
-
| `profiles/` | Personality profiles with `instructions.txt` + `tools.txt` + optional custom tools |
|
| 128 |
-
|
| 129 |
-
## Language Profile System
|
| 130 |
-
|
| 131 |
-
Six language tutor profiles available in `profiles/`:
|
| 132 |
-
- **`default`**: Generic language partner that adapts to any language
|
| 133 |
-
- **`french_tutor`**: Delphine, a French conversation partner with cultural context
|
| 134 |
-
- **`spanish_tutor`**: Sofia, a Mexican Spanish conversation partner
|
| 135 |
-
- **`german_tutor`**: Lukas, a German tutor teaching Standard German (Hochdeutsch)
|
| 136 |
-
- **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
|
| 137 |
-
- **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
|
| 138 |
-
|
| 139 |
-
Each profile in `reachy_mini_language_tutor/profiles/<name>/` contains:
|
| 140 |
-
- **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
|
| 141 |
-
- **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
|
| 142 |
-
- **`proactive.txt`**: Set to `true` for proactive greeting mode
|
| 143 |
-
- **`language.txt`**: ISO language code for transcription (e.g., `es`, `fr`)
|
| 144 |
-
- **`voice.txt`**: Voice name (e.g., `coral`, `sage`)
|
| 145 |
-
- **Optional Python files**: Custom tool implementations (subclass `Tool` from `tools/core_tools.py`)
|
| 146 |
-
|
| 147 |
-
### Template-Based Instruction Design
|
| 148 |
-
|
| 149 |
-
Tutor `instructions.txt` files compose from **shared prompts** (in `prompts/language_tutoring/`) + **unique content**:
|
| 150 |
-
|
| 151 |
-
**Shared prompts** (13 files, ~165 lines total):
|
| 152 |
-
- `[language_tutoring/proactive_engagement]` - Session start, memory recall, collecting personal info
|
| 153 |
-
- `[language_tutoring/language_behavior]` - English-first instruction approach
|
| 154 |
-
- `[language_tutoring/adaptive_support]` - Detecting and responding to learner struggle
|
| 155 |
-
- `[language_tutoring/correction_style]` - Recasting and error handling
|
| 156 |
-
- `[language_tutoring/grammar_explanation_structure]` - Framework for "why?" deep-dives
|
| 157 |
-
- `[language_tutoring/conversation_topics]` - Topic guidance
|
| 158 |
-
- `[language_tutoring/robot_expressiveness]` - Using robot capabilities for teaching
|
| 159 |
-
- `[language_tutoring/response_guidelines]` - Response format guidance
|
| 160 |
-
- `[language_tutoring/vocabulary_teaching]` - New word introduction pattern
|
| 161 |
-
- `[language_tutoring/memory_usage]` - Recall/remember tool usage
|
| 162 |
-
- `[language_tutoring/error_pattern_tracking]` - Specific error tracking with context
|
| 163 |
-
- `[language_tutoring/session_wrap_up]` - End-of-session summary protocol
|
| 164 |
-
- `[language_tutoring/final_notes]` - Core teaching philosophy
|
| 165 |
-
|
| 166 |
-
**Unique content per tutor** (~100-150 lines):
|
| 167 |
-
- IDENTITY (tutor name, personality, background)
|
| 168 |
-
- Grammar explanation example (language-specific concept)
|
| 169 |
-
- Language-specific teaching approach (e.g., MEXICAN SPANISH SPECIFICS)
|
| 170 |
-
- Example interactions (dialogue in target language)
|
| 171 |
-
- Cultural topics (region-specific insights)
|
| 172 |
-
|
| 173 |
-
**Benefits**: Tutors are 64-85% smaller. Shared methodology updates propagate to all tutors automatically. New language profiles focus only on unique content.
|
| 174 |
-
|
| 175 |
-
Load profiles via:
|
| 176 |
-
- CLI: `--profile <name>`
|
| 177 |
-
- Environment: `REACHY_MINI_CUSTOM_PROFILE=<name>` in `.env`
|
| 178 |
-
- Gradio UI: Select and hot-reload instructions (tools require restart)
|
| 179 |
-
|
| 180 |
-
## Enhanced Feedback System
|
| 181 |
-
|
| 182 |
-
The tutors implement a multi-layered feedback approach designed for effective language learning:
|
| 183 |
-
|
| 184 |
-
### Grammar Explanation Mode
|
| 185 |
-
When learners ask "why?", "explain that", or show confusion, tutors:
|
| 186 |
-
1. Pause practice entirely
|
| 187 |
-
2. Switch to full English explanation mode
|
| 188 |
-
3. Provide complete grammar lessons with rules, examples, and memory tricks
|
| 189 |
-
4. Have learners practice the concept immediately
|
| 190 |
-
5. Store the explanation in memory for future reference
|
| 191 |
-
|
| 192 |
-
Each tutor includes language-specific examples:
|
| 193 |
-
- **French**: Passé composé with être, DR MRS VANDERTRAMP mnemonic
|
| 194 |
-
- **Spanish**: Ser vs estar distinction, preterite vs imperfect
|
| 195 |
-
- **German**: Akkusativ case changes, "AKK-use" mnemonic for direct objects
|
| 196 |
-
- **Italian**: Gender agreement with O/A/I/E ending patterns
|
| 197 |
-
- **Portuguese**: Ser vs estar, gerund usage (Brazilian vs European)
|
| 198 |
-
|
| 199 |
-
### Error Pattern Tracking
|
| 200 |
-
Tutors store errors with specific context using the `remember` tool (category: `struggle`):
|
| 201 |
-
- **Specificity**: "Confused 'ser' vs 'estar' describing emotions" not just "verb issues"
|
| 202 |
-
- **Context**: "Used 'j'ai allé' instead of 'je suis allé' in past tense narrative"
|
| 203 |
-
- **Frequency**: "Third time confusing gender of 'table'"
|
| 204 |
-
|
| 205 |
-
At session start, tutors recall past struggles and incorporate review naturally.
|
| 206 |
-
|
| 207 |
-
### Session Summaries
|
| 208 |
-
When sessions end, tutors provide spoken recaps:
|
| 209 |
-
1. Topics/vocabulary covered
|
| 210 |
-
2. One highlight (something done well)
|
| 211 |
-
3. One area to focus on next time
|
| 212 |
-
4. Store summary in memory (category: `progress`)
|
| 213 |
-
5. End with encouragement and celebratory dance
|
| 214 |
-
|
| 215 |
-
## Configuration (.env)
|
| 216 |
-
|
| 217 |
-
Required:
|
| 218 |
-
```
|
| 219 |
-
OPENAI_API_KEY=your_key_here
|
| 220 |
-
```
|
| 221 |
-
|
| 222 |
-
Optional:
|
| 223 |
-
```
|
| 224 |
-
REACHY_MINI_CUSTOM_PROFILE=french_tutor # Language profile to load
|
| 225 |
-
SUPERMEMORY_API_KEY=your_key_here # Persistent memory for tutors
|
| 226 |
-
MODEL_NAME=gpt-realtime # Override realtime model
|
| 227 |
-
HF_HOME=./cache # Local VLM cache (--local-vision)
|
| 228 |
-
HF_TOKEN=your_hf_token # For Hugging Face models/emotions
|
| 229 |
-
LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model path
|
| 230 |
-
```
|
| 231 |
-
|
| 232 |
-
**Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
|
| 233 |
-
|
| 234 |
-
## Available LLM Tools
|
| 235 |
-
|
| 236 |
-
| Tool | Action | Dependencies |
|
| 237 |
-
|------|--------|--------------|
|
| 238 |
-
| `move_head` | Queue head pose change (left/right/up/down/front) | Core |
|
| 239 |
-
| `camera` | Capture frame and send to vision model | Camera worker |
|
| 240 |
-
| `head_tracking` | Enable/disable face-tracking offsets (not facial recognition) | Camera + head tracker |
|
| 241 |
-
| `dance` | Queue dance from `reachy_mini_dances_library` | Core |
|
| 242 |
-
| `stop_dance` | Clear dance queue | Core |
|
| 243 |
-
| `play_emotion` | Play recorded emotion clip | Core + `HF_TOKEN` |
|
| 244 |
-
| `stop_emotion` | Clear emotion queue | Core |
|
| 245 |
-
| `do_nothing` | Explicitly remain idle | Core |
|
| 246 |
-
| `recall` | Search persistent memory for learner information | `SUPERMEMORY_API_KEY` |
|
| 247 |
-
| `remember` | Store observations about learner for future sessions | `SUPERMEMORY_API_KEY` |
|
| 248 |
-
|
| 249 |
-
## Code Style Conventions
|
| 250 |
-
|
| 251 |
-
- **Type checking**: Strict mypy enabled (`mypy reachy_mini_language_tutor/`)
|
| 252 |
-
- **Formatting**: Ruff with 119-char line length
|
| 253 |
-
- **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
|
| 254 |
-
- **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
|
| 255 |
-
- **Quote style**: Double quotes
|
| 256 |
-
- **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
|
| 257 |
-
|
| 258 |
-
## Development Tips
|
| 259 |
-
|
| 260 |
-
### Adding Custom Tools
|
| 261 |
-
1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`)
|
| 262 |
-
2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
|
| 263 |
-
3. Implement `name`, `description`, `parameters`, and `__call__()` method
|
| 264 |
-
4. Add tool name to `profiles/<profile_name>/tools.txt`
|
| 265 |
-
5. See `profiles/french_tutor/recall.py` and `profiles/french_tutor/remember.py` for examples
|
| 266 |
-
|
| 267 |
-
### Motion Control Principles
|
| 268 |
-
- **100Hz loop is sacred** - never block the `MovementManager` thread
|
| 269 |
-
- Offload heavy work to separate threads/processes
|
| 270 |
-
- Primary moves are queued and executed sequentially
|
| 271 |
-
- Secondary offsets are blended additively in real-time
|
| 272 |
-
- Use `BreathingMove` as idle fallback when queue is empty
|
| 273 |
-
|
| 274 |
-
**Critical SDK Constraints:**
|
| 275 |
-
- **65° Yaw Delta Limit**: Head yaw and body yaw cannot differ by more than 65°. The SDK auto-clamps violations, which means aggressive head tracking or wobble offsets may be silently limited if they push the combined pose beyond this constraint.
|
| 276 |
-
- **set_target() vs goto_target()**: This app uses `set_target()` in the 100Hz loop because it bypasses interpolation, allowing real-time composition of primary + secondary poses. Using `goto_target()` would conflict with the manual control loop since it runs its own interpolation.
|
| 277 |
-
|
| 278 |
-
### Thread Safety
|
| 279 |
-
- `CameraWorker` uses locks for frame buffer and tracking offsets
|
| 280 |
-
- `MovementManager` uses queue for command dispatch
|
| 281 |
-
- Never share mutable state between threads without synchronization
|
| 282 |
-
|
| 283 |
-
### Vision Processing
|
| 284 |
-
- Default vision goes through OpenAI's gpt-realtime (when camera tool is called)
|
| 285 |
-
- Local vision (`--local-vision`) runs SmolVLM2 periodically on-device
|
| 286 |
-
- Face tracking is separate from vision - it only tracks face position for head offsets (not recognition)
|
| 287 |
-
|
| 288 |
-
### Working with Gradio UI
|
| 289 |
-
|
| 290 |
-
**Architecture**:
|
| 291 |
-
- Gradio UI runs in the main thread, launched by `main.py`
|
| 292 |
-
- Provides web interface at `http://127.0.0.1:7860/` (default port)
|
| 293 |
-
- Runs alongside robot control threads (does not block motion control)
|
| 294 |
-
- Two main UI components:
|
| 295 |
-
- `console.py`: `LocalStream` class handles audio I/O and settings routes for headless/console mode
|
| 296 |
-
- `gradio_personality.py`: `PersonalityUI` class manages profile selection and live instruction reloading
|
| 297 |
-
|
| 298 |
-
**Key UI Features**:
|
| 299 |
-
- **Live audio streaming**: Bidirectional audio via Gradio's audio components
|
| 300 |
-
- **Profile management**: Hot-reload personality instructions without restart (tools require restart)
|
| 301 |
-
- **Settings routes**: Dynamic endpoints for UI configuration and state
|
| 302 |
-
- **Real-time feedback**: Status updates and conversation display
|
| 303 |
-
|
| 304 |
-
**Development Guidelines**:
|
| 305 |
-
- **State management**: Gradio components should not hold critical state - use `MovementManager`, `OpenaiRealtimeHandler`, or `Config` as source of truth
|
| 306 |
-
- **Thread safety**: UI callbacks may run in separate threads - always use proper synchronization when accessing shared resources
|
| 307 |
-
- **Blocking operations**: Never perform long-running operations directly in Gradio callbacks - offload to background threads/queues
|
| 308 |
-
- **Hot-reloading**: Changes to Gradio UI code require app restart (unlike profile instructions which can be hot-loaded)
|
| 309 |
-
- **Testing**: Test UI locally with `--gradio` flag before deploying changes
|
| 310 |
-
|
| 311 |
-
**Common Patterns**:
|
| 312 |
-
- Use `gr.Audio()` with streaming for real-time audio I/O
|
| 313 |
-
- Use `gr.Dropdown()` for profile selection with dynamic refresh
|
| 314 |
-
- Use `gr.Button()` callbacks to trigger actions via `MovementManager` queue
|
| 315 |
-
- Use `gr.Textbox()` with `interactive=True` for live instruction editing
|
| 316 |
-
- Return updates via component `.update()` methods for reactive UI
|
| 317 |
-
|
| 318 |
-
**Gradio-Specific Considerations**:
|
| 319 |
-
- Gradio apps auto-reload on file changes in debug mode (use `debug=True` in `gr.Interface.launch()`)
|
| 320 |
-
- Share links (`share=True`) create public tunnels - avoid for production
|
| 321 |
-
- Custom CSS/themes can be applied via `gr.themes` or custom CSS strings
|
| 322 |
-
- Component visibility and interactivity can be toggled dynamically via `.update()`
|
| 323 |
-
|
| 324 |
-
**Debugging**:
|
| 325 |
-
- Check browser console for JavaScript errors
|
| 326 |
-
- Use `print()` statements in callbacks (visible in terminal output)
|
| 327 |
-
- Gradio exceptions are caught and displayed in UI - check both UI and terminal
|
| 328 |
-
- Audio issues: Verify browser permissions for microphone/speaker access
|
| 329 |
-
|
| 330 |
-
## Testing
|
| 331 |
-
- Tests located in `tests/`
|
| 332 |
-
- Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
|
| 333 |
-
- Audio fixtures available in `conftest.py`
|
| 334 |
-
- Run with `pytest` after `uv sync --group dev`
|
| 335 |
-
|
| 336 |
-
## Dependencies & Extras
|
| 337 |
-
|
| 338 |
-
| Extra | Purpose | Key Packages |
|
| 339 |
-
|-------|---------|--------------|
|
| 340 |
-
| Base | Core audio/vision/motion | `fastrtc`, `aiortc`, `openai`, `gradio`, `opencv-python`, `reachy_mini` |
|
| 341 |
-
| `reachy_mini_wireless` | Wireless robot support | `PyGObject`, `gst-signalling` (GStreamer) |
|
| 342 |
-
| `local_vision` | Local VLM processing | `torch`, `transformers`, `num2words` |
|
| 343 |
-
| `yolo_vision` | YOLO head tracking | `ultralytics`, `supervision` |
|
| 344 |
-
| `mediapipe_vision` | MediaPipe tracking | `mediapipe==0.10.14` |
|
| 345 |
-
| `all_vision` | All vision features | Combination of above |
|
| 346 |
-
| `dev` | Development tools | `pytest`, `ruff`, `mypy`, `pre-commit` |
|
| 347 |
-
|
| 348 |
-
## Common Troubleshooting
|
| 349 |
-
|
| 350 |
-
**TimeoutError on startup**: Reachy Mini daemon not running. Install and start Reachy Mini SDK.
|
| 351 |
-
|
| 352 |
-
**Vision not working**: Check that camera is connected and accessible. Use `--no-camera` to disable if not needed.
|
| 353 |
-
|
| 354 |
-
**Wireless connection issues**: Ensure `--wireless-version` flag is used and daemon started with same flag. Requires `reachy_mini_wireless` extra.
|
| 355 |
-
|
| 356 |
-
**Local vision slow**: SmolVLM2 benefits from GPU/MPS acceleration. Consider using default gpt-realtime vision if CPU-only.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
CLAUDE.md
|