Matt Vaughn Claude Sonnet 4.5 commited on
Commit
1af9e08
·
1 Parent(s): 87b31d1

Docs: Update CLAUDE.md with recent improvements and enforce uv run usage

Browse files

## Changes

**CLAUDE.md updates:**
- Add "Recent Improvements" section documenting 6 major changes from Dec 2025 - Jan 2026
- Update all development workflow commands to use `uv run` prefix
- Fix path references: `src/reachy_mini_language_tutor/` → `reachy_mini_language_tutor/`
- Add privacy protections documentation for persistent memory
- Update tool documentation to reflect recall/remember moved to shared tools/
- Emphasize `uv run` for ruff, mypy, pytest in Code Style Conventions

**GEMINI.md:**
- Replace duplicate file with symlink to CLAUDE.md for consistency

Improvements documented:
- Privacy protections for persistent memory (Jan 2026)
- Normalized language tutor profiles with English-first approach (Jan 2026)
- Shared memory tools eliminating 455 lines of duplication (Jan 2026)
- Source code migration to root directory (Dec 2025)
- SDK compatibility checks (Jan 2026)
- Template-based tutor design with 64-85% size reduction (Dec 2025)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Files changed (3) hide show
  1. CLAUDE.md +36 -11
  2. GEMINI.md +0 -356
  3. GEMINI.md +1 -0
CLAUDE.md CHANGED
@@ -15,6 +15,20 @@ Reachy Language Partner - a language learning companion for the Reachy Mini robo
15
 
16
  Powered by OpenAI's realtime API, vision processing, and choreographed motion.
17
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  ## Important Resources
19
 
20
  **ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
@@ -58,12 +72,14 @@ reachy-mini-language-tutor --profile <name> # Load custom profile
58
  ```
59
 
60
  ### Development Workflow
 
 
61
  ```bash
62
- ruff check . # Lint and format check
63
- ruff format . # Auto-format code
64
- mypy src/ # Type checking (strict mode enabled)
65
- pytest # Run test suite
66
- pytest tests/test_openai_realtime.py # Run specific test file
67
  ```
68
 
69
  ## Architecture
@@ -136,7 +152,7 @@ Six language tutor profiles available in `profiles/`:
136
  - **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
137
  - **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
138
 
139
- Each profile in `src/reachy_mini_language_tutor/profiles/<name>/` contains:
140
  - **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
141
  - **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
142
  - **`proactive.txt`**: Set to `true` for proactive greeting mode
@@ -231,8 +247,15 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
231
 
232
  **Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
233
 
 
 
 
 
 
234
  ## Available LLM Tools
235
 
 
 
236
  | Tool | Action | Dependencies |
237
  |------|--------|--------------|
238
  | `move_head` | Queue head pose change (left/right/up/down/front) | Core |
@@ -248,21 +271,22 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
248
 
249
  ## Code Style Conventions
250
 
251
- - **Type checking**: Strict mypy enabled (`mypy src/`)
252
- - **Formatting**: Ruff with 119-char line length
253
  - **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
254
  - **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
255
  - **Quote style**: Double quotes
256
  - **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
 
257
 
258
  ## Development Tips
259
 
260
  ### Adding Custom Tools
261
- 1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`)
262
  2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
263
  3. Implement `name`, `description`, `parameters`, and `__call__()` method
264
  4. Add tool name to `profiles/<profile_name>/tools.txt`
265
- 5. See `profiles/french_tutor/recall.py` and `profiles/french_tutor/remember.py` for examples
266
 
267
  ### Motion Control Principles
268
  - **100Hz loop is sacred** - never block the `MovementManager` thread
@@ -331,7 +355,8 @@ LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model pa
331
  - Tests located in `tests/`
332
  - Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
333
  - Audio fixtures available in `conftest.py`
334
- - Run with `pytest` after `uv sync --group dev`
 
335
 
336
  ## Dependencies & Extras
337
 
 
15
 
16
  Powered by OpenAI's realtime API, vision processing, and choreographed motion.
17
 
18
+ ## Recent Improvements
19
+
20
+ **Privacy Protections (Jan 2026)**: Added comprehensive privacy guidelines to persistent memory system. Memory now implements moderate privacy level with clear allowed/excluded data categories.
21
+
22
+ **Normalized Language Tutor Profiles (Jan 2026)**: All language tutors now follow consistent English-first instruction approach. Cultural content is properly framed within the teaching methodology rather than being a primary focus.
23
+
24
+ **Shared Memory Tools (Jan 2026)**: Moved `recall` and `remember` tools from profile-specific implementations to shared `tools/` directory, eliminating 455 lines of duplicate code and fixing "unknown tool" errors for default profile.
25
+
26
+ **Source Code Migration (Dec 2025)**: Migrated source code from `src/reachy_mini_language_tutor/` to root-level `reachy_mini_language_tutor/` directory for cleaner project structure.
27
+
28
+ **SDK Compatibility Check (Jan 2026)**: Added defensive version checking for SDK methods (e.g., `clear_output_buffer()`), preventing crashes when using different SDK versions.
29
+
30
+ **Template-Based Tutor Design (Dec 2025)**: Extracted 13 shared prompt components (~165 lines) into reusable templates, reducing tutor file sizes by 64-85% and ensuring consistent methodology across all language tutors.
31
+
32
  ## Important Resources
33
 
34
  **ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
 
72
  ```
73
 
74
  ### Development Workflow
75
+ **IMPORTANT: Always use `uv run` to execute development tools to ensure they run in the correct virtual environment.**
76
+
77
  ```bash
78
+ uv run ruff check . # Lint and format check
79
+ uv run ruff format . # Auto-format code
80
+ uv run mypy reachy_mini_language_tutor/ # Type checking (strict mode enabled)
81
+ uv run pytest # Run test suite
82
+ uv run pytest tests/test_openai_realtime.py # Run specific test file
83
  ```
84
 
85
  ## Architecture
 
152
  - **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
153
  - **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
154
 
155
+ Each profile in `reachy_mini_language_tutor/profiles/<name>/` contains:
156
  - **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
157
  - **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
158
  - **`proactive.txt`**: Set to `true` for proactive greeting mode
 
247
 
248
  **Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
249
 
250
+ **Privacy Protections**: The memory system implements moderate privacy guidelines:
251
+ - **ALLOWED**: First names, general region, occupation category, interests, learning data
252
+ - **EXCLUDED**: Age, specific location, family names, sensitive details, travel dates
253
+ - Implicit consent model via `SUPERMEMORY_API_KEY` configuration
254
+
255
  ## Available LLM Tools
256
 
257
+ **Note**: The `recall` and `remember` tools are now in the shared `tools/` directory (as of recent refactor), making them available to all profiles without duplication.
258
+
259
  | Tool | Action | Dependencies |
260
  |------|--------|--------------|
261
  | `move_head` | Queue head pose change (left/right/up/down/front) | Core |
 
271
 
272
  ## Code Style Conventions
273
 
274
+ - **Type checking**: Strict mypy enabled (`uv run mypy reachy_mini_language_tutor/`)
275
+ - **Formatting**: Ruff with 119-char line length (`uv run ruff format .`)
276
  - **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
277
  - **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
278
  - **Quote style**: Double quotes
279
  - **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
280
+ - **Tool execution**: Always use `uv run` for ruff, mypy, pytest, and other dev tools
281
 
282
  ## Development Tips
283
 
284
  ### Adding Custom Tools
285
+ 1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`) for profile-specific tools, or in `tools/` for shared tools
286
  2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
287
  3. Implement `name`, `description`, `parameters`, and `__call__()` method
288
  4. Add tool name to `profiles/<profile_name>/tools.txt`
289
+ 5. See `tools/recall.py` and `tools/remember.py` for examples of shared tools available to all profiles
290
 
291
  ### Motion Control Principles
292
  - **100Hz loop is sacred** - never block the `MovementManager` thread
 
355
  - Tests located in `tests/`
356
  - Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
357
  - Audio fixtures available in `conftest.py`
358
+ - Run with `uv run pytest` after `uv sync --group dev`
359
+ - Always use `uv run pytest` to ensure tests run in the correct environment
360
 
361
  ## Dependencies & Extras
362
 
GEMINI.md DELETED
@@ -1,356 +0,0 @@
1
- # GEMINI.md
2
-
3
- This file provides guidance to Gemini Code (google.ai/gemini/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
- Reachy Language Partner - a language learning companion for the Reachy Mini robot. Practice conversational skills in French, Spanish, German, Italian, Portuguese, and other languages through natural dialogue with an expressive robot partner.
7
-
8
- **Key Features:**
9
- - Persistent memory across sessions (tracks progress, struggles, preferences)
10
- - Proactive engagement and gentle correction through recasting
11
- - Grammar deep-dive mode with complete explanations on demand
12
- - Error pattern tracking with proactive review at session start
13
- - Session summaries with highlights and next-steps recommendations
14
- - Expressive robot feedback (dances, emotions, celebrations)
15
-
16
- Powered by OpenAI's realtime API, vision processing, and choreographed motion.
17
-
18
- ## Important Resources
19
-
20
- **ALWAYS consult the Reachy Mini Python SDK documentation for canonical information about robot implementation and design:**
21
- - **SDK Documentation**: https://github.com/pollen-robotics/reachy_mini/blob/develop/docs/SDK/readme.md
22
-
23
- When working on robot control, motion systems, hardware interfaces, or any Reachy Mini-specific functionality, refer to the official SDK documentation first. This is the authoritative source for:
24
- - Robot API reference and methods
25
- - Hardware capabilities and limitations
26
- - Control loop best practices
27
- - Joint limits and coordinate systems
28
- - Official code examples and patterns
29
-
30
- ## Build & Run Commands
31
-
32
- ### Installation
33
- ```bash
34
- # Using uv (recommended)
35
- uv venv --python 3.12.1
36
- source .venv/bin/activate
37
- uv sync # Base install
38
- uv sync --extra all_vision # With all vision features
39
- uv sync --extra reachy_mini_wireless # For wireless robot support
40
- uv sync --group dev # Add dev tools (pytest, ruff, mypy)
41
-
42
- # Using pip
43
- python -m venv .venv
44
- source .venv/bin/activate
45
- pip install -e .
46
- pip install -e .[all_vision,dev]
47
- ```
48
-
49
- ### Running the App
50
- ```bash
51
- reachy-mini-language-tutor # Console mode (default)
52
- reachy-mini-language-tutor --gradio # Web UI at http://127.0.0.1:7860/
53
- reachy-mini-language-tutor --head-tracker mediapipe # With face tracking
54
- reachy-mini-language-tutor --local-vision # Local SmolVLM2 vision
55
- reachy-mini-language-tutor --wireless-version # For wireless robot
56
- reachy-mini-language-tutor --no-camera # Audio-only mode
57
- reachy-mini-language-tutor --profile <name> # Load custom profile
58
- ```
59
-
60
- ### Development Workflow
61
- ```bash
62
- ruff check . # Lint and format check
63
- ruff format . # Auto-format code
64
- mypy reachy_mini_language_tutor/ # Type checking (strict mode enabled)
65
- pytest # Run test suite
66
- pytest tests/test_openai_realtime.py # Run specific test file
67
- ```
68
-
69
- ## Architecture
70
-
71
- ### Layered Motion System (Core Design)
72
- The robot uses a **compose-based motion blending** system with primary and secondary layers:
73
-
74
- **Primary Moves** (sequential, mutually exclusive queue in `MovementManager`):
75
- - Dances (from `reachy_mini_dances_library`)
76
- - Recorded emotions (Hugging Face assets)
77
- - Goto poses (head positioning)
78
- - Breathing (idle state fallback)
79
-
80
- **Secondary Offsets** (additive, blended in real-time):
81
- - Speech wobble (`HeadWobbler`) - reactive to audio stream activity
82
- - Face tracking offsets (`CameraWorker`) - smooth interpolation with 2s fade-out when face is lost
83
-
84
- **Control Loop**: `MovementManager` runs at **100Hz** in a dedicated thread, composing primary + secondary poses and calling `robot.set_target()`.
85
-
86
- ### Threading Model
87
- - **Main thread**: Launches Gradio/console UI
88
- - **MovementManager thread**: 100Hz motion control loop
89
- - **HeadWobbler thread**: Audio-reactive motion processing (speech detection)
90
- - **CameraWorker thread**: 30Hz+ frame capture + face tracking
91
- - **VisionManager thread**: Periodic local VLM inference (if `--local-vision` enabled)
92
-
93
- ### Tool Dispatch Pattern
94
- ```
95
- User audio → OpenaiRealtimeHandler (24kHz mono WebRTC)
96
- → LLM generates tool calls
97
- → dispatch_tool_call() routes to Tool subclass
98
- → Tool enqueues command in MovementManager
99
- → MovementManager executes and returns status via AdditionalOutputs
100
- → Response audio + motion sent to robot/user
101
- ```
102
-
103
- ### Vision Pipeline Options
104
- - **Default (gpt-realtime)**: Camera tool sends frames to OpenAI for vision
105
- - **Local vision (`--local-vision`)**: VisionManager processes frames with SmolVLM2 (on-device CPU/GPU/MPS)
106
- - **Face tracking options**:
107
- - `--head-tracker yolo`: YOLOv8-based face detection (requires `yolo_vision` extra)
108
- - `--head-tracker mediapipe`: MediaPipe from `reachy_mini_toolbox` (requires `mediapipe_vision` extra, pinned to 0.10.14)
109
-
110
- ## Key File Responsibilities
111
-
112
- | File/Directory | Purpose |
113
- |----------------|---------|
114
- | `main.py` | Entry point - initializes robot, managers, UI (Gradio or console) |
115
- | `openai_realtime.py` | `OpenaiRealtimeHandler` - async WebRTC stream to OpenAI realtime API |
116
- | `moves.py` | `MovementManager` - 100Hz motion control loop with queue management |
117
- | `camera_worker.py` | `CameraWorker` - thread-safe frame buffering + face tracking offset calculation |
118
- | `audio/head_wobbler.py` | Speech-reactive head wobble (additive motion from audio stream) |
119
- | `tools/core_tools.py` | `Tool` base class + dispatch logic for LLM tool calls |
120
- | `tools/` | Built-in tools: `move_head`, `dance`, `play_emotion`, `camera`, `head_tracking`, etc. |
121
- | `config.py` | `Config` class - loads `.env` and manages profile settings |
122
- | `prompts.py` | Prompt loading with `[placeholder]` syntax for reusable components |
123
- | `console.py` | `LocalStream` - headless mode with direct audio I/O + settings routes |
124
- | `gradio_personality.py` | `PersonalityUI` - profile selector, creator, live instruction reload |
125
- | `vision/processors.py` | Local vision manager (SmolVLM2 initialization) |
126
- | `vision/yolo_head_tracker.py` | YOLO-based face detection for head tracking |
127
- | `profiles/` | Personality profiles with `instructions.txt` + `tools.txt` + optional custom tools |
128
-
129
- ## Language Profile System
130
-
131
- Six language tutor profiles available in `profiles/`:
132
- - **`default`**: Generic language partner that adapts to any language
133
- - **`french_tutor`**: Delphine, a French conversation partner with cultural context
134
- - **`spanish_tutor`**: Sofia, a Mexican Spanish conversation partner
135
- - **`german_tutor`**: Lukas, a German tutor teaching Standard German (Hochdeutsch)
136
- - **`italian_tutor`**: Chiara, an Italian tutor from Florence with cultural insights
137
- - **`portuguese_tutor`**: Rafael, a Brazilian Portuguese tutor from São Paulo
138
-
139
- Each profile in `reachy_mini_language_tutor/profiles/<name>/` contains:
140
- - **`instructions.txt`**: System prompt (uses `[placeholder]` syntax to compose from shared + unique content)
141
- - **`tools.txt`**: Enabled tools list (comment with `#`, one per line)
142
- - **`proactive.txt`**: Set to `true` for proactive greeting mode
143
- - **`language.txt`**: ISO language code for transcription (e.g., `es`, `fr`)
144
- - **`voice.txt`**: Voice name (e.g., `coral`, `sage`)
145
- - **Optional Python files**: Custom tool implementations (subclass `Tool` from `tools/core_tools.py`)
146
-
147
- ### Template-Based Instruction Design
148
-
149
- Tutor `instructions.txt` files compose from **shared prompts** (in `prompts/language_tutoring/`) + **unique content**:
150
-
151
- **Shared prompts** (13 files, ~165 lines total):
152
- - `[language_tutoring/proactive_engagement]` - Session start, memory recall, collecting personal info
153
- - `[language_tutoring/language_behavior]` - English-first instruction approach
154
- - `[language_tutoring/adaptive_support]` - Detecting and responding to learner struggle
155
- - `[language_tutoring/correction_style]` - Recasting and error handling
156
- - `[language_tutoring/grammar_explanation_structure]` - Framework for "why?" deep-dives
157
- - `[language_tutoring/conversation_topics]` - Topic guidance
158
- - `[language_tutoring/robot_expressiveness]` - Using robot capabilities for teaching
159
- - `[language_tutoring/response_guidelines]` - Response format guidance
160
- - `[language_tutoring/vocabulary_teaching]` - New word introduction pattern
161
- - `[language_tutoring/memory_usage]` - Recall/remember tool usage
162
- - `[language_tutoring/error_pattern_tracking]` - Specific error tracking with context
163
- - `[language_tutoring/session_wrap_up]` - End-of-session summary protocol
164
- - `[language_tutoring/final_notes]` - Core teaching philosophy
165
-
166
- **Unique content per tutor** (~100-150 lines):
167
- - IDENTITY (tutor name, personality, background)
168
- - Grammar explanation example (language-specific concept)
169
- - Language-specific teaching approach (e.g., MEXICAN SPANISH SPECIFICS)
170
- - Example interactions (dialogue in target language)
171
- - Cultural topics (region-specific insights)
172
-
173
- **Benefits**: Tutors are 64-85% smaller. Shared methodology updates propagate to all tutors automatically. New language profiles focus only on unique content.
174
-
175
- Load profiles via:
176
- - CLI: `--profile <name>`
177
- - Environment: `REACHY_MINI_CUSTOM_PROFILE=<name>` in `.env`
178
- - Gradio UI: Select and hot-reload instructions (tools require restart)
179
-
180
- ## Enhanced Feedback System
181
-
182
- The tutors implement a multi-layered feedback approach designed for effective language learning:
183
-
184
- ### Grammar Explanation Mode
185
- When learners ask "why?", "explain that", or show confusion, tutors:
186
- 1. Pause practice entirely
187
- 2. Switch to full English explanation mode
188
- 3. Provide complete grammar lessons with rules, examples, and memory tricks
189
- 4. Have learners practice the concept immediately
190
- 5. Store the explanation in memory for future reference
191
-
192
- Each tutor includes language-specific examples:
193
- - **French**: Passé composé with être, DR MRS VANDERTRAMP mnemonic
194
- - **Spanish**: Ser vs estar distinction, preterite vs imperfect
195
- - **German**: Akkusativ case changes, "AKK-use" mnemonic for direct objects
196
- - **Italian**: Gender agreement with O/A/I/E ending patterns
197
- - **Portuguese**: Ser vs estar, gerund usage (Brazilian vs European)
198
-
199
- ### Error Pattern Tracking
200
- Tutors store errors with specific context using the `remember` tool (category: `struggle`):
201
- - **Specificity**: "Confused 'ser' vs 'estar' describing emotions" not just "verb issues"
202
- - **Context**: "Used 'j'ai allé' instead of 'je suis allé' in past tense narrative"
203
- - **Frequency**: "Third time confusing gender of 'table'"
204
-
205
- At session start, tutors recall past struggles and incorporate review naturally.
206
-
207
- ### Session Summaries
208
- When sessions end, tutors provide spoken recaps:
209
- 1. Topics/vocabulary covered
210
- 2. One highlight (something done well)
211
- 3. One area to focus on next time
212
- 4. Store summary in memory (category: `progress`)
213
- 5. End with encouragement and celebratory dance
214
-
215
- ## Configuration (.env)
216
-
217
- Required:
218
- ```
219
- OPENAI_API_KEY=your_key_here
220
- ```
221
-
222
- Optional:
223
- ```
224
- REACHY_MINI_CUSTOM_PROFILE=french_tutor # Language profile to load
225
- SUPERMEMORY_API_KEY=your_key_here # Persistent memory for tutors
226
- MODEL_NAME=gpt-realtime # Override realtime model
227
- HF_HOME=./cache # Local VLM cache (--local-vision)
228
- HF_TOKEN=your_hf_token # For Hugging Face models/emotions
229
- LOCAL_VISION_MODEL=HuggingFaceTB/SmolVLM2-2.2B-Instruct # Local vision model path
230
- ```
231
-
232
- **Persistent Memory**: `SUPERMEMORY_API_KEY` enables cross-session memory via [supermemory.ai](https://supermemory.ai). When configured, tutors remember learner names, skill levels, error patterns, and progress. Powers the `recall` and `remember` tools.
233
-
234
- ## Available LLM Tools
235
-
236
- | Tool | Action | Dependencies |
237
- |------|--------|--------------|
238
- | `move_head` | Queue head pose change (left/right/up/down/front) | Core |
239
- | `camera` | Capture frame and send to vision model | Camera worker |
240
- | `head_tracking` | Enable/disable face-tracking offsets (not facial recognition) | Camera + head tracker |
241
- | `dance` | Queue dance from `reachy_mini_dances_library` | Core |
242
- | `stop_dance` | Clear dance queue | Core |
243
- | `play_emotion` | Play recorded emotion clip | Core + `HF_TOKEN` |
244
- | `stop_emotion` | Clear emotion queue | Core |
245
- | `do_nothing` | Explicitly remain idle | Core |
246
- | `recall` | Search persistent memory for learner information | `SUPERMEMORY_API_KEY` |
247
- | `remember` | Store observations about learner for future sessions | `SUPERMEMORY_API_KEY` |
248
-
249
- ## Code Style Conventions
250
-
251
- - **Type checking**: Strict mypy enabled (`mypy reachy_mini_language_tutor/`)
252
- - **Formatting**: Ruff with 119-char line length
253
- - **Docstrings**: Required for all public functions/classes (ruff `D` rules enabled)
254
- - **Import sorting**: `isort` via ruff (local-folder: `reachy_mini_language_tutor`)
255
- - **Quote style**: Double quotes
256
- - **Async patterns**: Use `asyncio` for OpenAI realtime API, threading for robot control
257
-
258
- ## Development Tips
259
-
260
- ### Adding Custom Tools
261
- 1. Create Python file in `profiles/<profile_name>/` (e.g., `my_tool.py`)
262
- 2. Subclass `reachy_mini_language_tutor.tools.core_tools.Tool`
263
- 3. Implement `name`, `description`, `parameters`, and `__call__()` method
264
- 4. Add tool name to `profiles/<profile_name>/tools.txt`
265
- 5. See `profiles/french_tutor/recall.py` and `profiles/french_tutor/remember.py` for examples
266
-
267
- ### Motion Control Principles
268
- - **100Hz loop is sacred** - never block the `MovementManager` thread
269
- - Offload heavy work to separate threads/processes
270
- - Primary moves are queued and executed sequentially
271
- - Secondary offsets are blended additively in real-time
272
- - Use `BreathingMove` as idle fallback when queue is empty
273
-
274
- **Critical SDK Constraints:**
275
- - **65° Yaw Delta Limit**: Head yaw and body yaw cannot differ by more than 65°. The SDK auto-clamps violations, which means aggressive head tracking or wobble offsets may be silently limited if they push the combined pose beyond this constraint.
276
- - **set_target() vs goto_target()**: This app uses `set_target()` in the 100Hz loop because it bypasses interpolation, allowing real-time composition of primary + secondary poses. Using `goto_target()` would conflict with the manual control loop since it runs its own interpolation.
277
-
278
- ### Thread Safety
279
- - `CameraWorker` uses locks for frame buffer and tracking offsets
280
- - `MovementManager` uses queue for command dispatch
281
- - Never share mutable state between threads without synchronization
282
-
283
- ### Vision Processing
284
- - Default vision goes through OpenAI's gpt-realtime (when camera tool is called)
285
- - Local vision (`--local-vision`) runs SmolVLM2 periodically on-device
286
- - Face tracking is separate from vision - it only tracks face position for head offsets (not recognition)
287
-
288
- ### Working with Gradio UI
289
-
290
- **Architecture**:
291
- - Gradio UI runs in the main thread, launched by `main.py`
292
- - Provides web interface at `http://127.0.0.1:7860/` (default port)
293
- - Runs alongside robot control threads (does not block motion control)
294
- - Two main UI components:
295
- - `console.py`: `LocalStream` class handles audio I/O and settings routes for headless/console mode
296
- - `gradio_personality.py`: `PersonalityUI` class manages profile selection and live instruction reloading
297
-
298
- **Key UI Features**:
299
- - **Live audio streaming**: Bidirectional audio via Gradio's audio components
300
- - **Profile management**: Hot-reload personality instructions without restart (tools require restart)
301
- - **Settings routes**: Dynamic endpoints for UI configuration and state
302
- - **Real-time feedback**: Status updates and conversation display
303
-
304
- **Development Guidelines**:
305
- - **State management**: Gradio components should not hold critical state - use `MovementManager`, `OpenaiRealtimeHandler`, or `Config` as source of truth
306
- - **Thread safety**: UI callbacks may run in separate threads - always use proper synchronization when accessing shared resources
307
- - **Blocking operations**: Never perform long-running operations directly in Gradio callbacks - offload to background threads/queues
308
- - **Hot-reloading**: Changes to Gradio UI code require app restart (unlike profile instructions which can be hot-loaded)
309
- - **Testing**: Test UI locally with `--gradio` flag before deploying changes
310
-
311
- **Common Patterns**:
312
- - Use `gr.Audio()` with streaming for real-time audio I/O
313
- - Use `gr.Dropdown()` for profile selection with dynamic refresh
314
- - Use `gr.Button()` callbacks to trigger actions via `MovementManager` queue
315
- - Use `gr.Textbox()` with `interactive=True` for live instruction editing
316
- - Return updates via component `.update()` methods for reactive UI
317
-
318
- **Gradio-Specific Considerations**:
319
- - Gradio apps auto-reload on file changes in debug mode (use `debug=True` in `gr.Interface.launch()`)
320
- - Share links (`share=True`) create public tunnels - avoid for production
321
- - Custom CSS/themes can be applied via `gr.themes` or custom CSS strings
322
- - Component visibility and interactivity can be toggled dynamically via `.update()`
323
-
324
- **Debugging**:
325
- - Check browser console for JavaScript errors
326
- - Use `print()` statements in callbacks (visible in terminal output)
327
- - Gradio exceptions are caught and displayed in UI - check both UI and terminal
328
- - Audio issues: Verify browser permissions for microphone/speaker access
329
-
330
- ## Testing
331
- - Tests located in `tests/`
332
- - Key test: `test_openai_realtime.py` (AsyncStreamHandler tests)
333
- - Audio fixtures available in `conftest.py`
334
- - Run with `pytest` after `uv sync --group dev`
335
-
336
- ## Dependencies & Extras
337
-
338
- | Extra | Purpose | Key Packages |
339
- |-------|---------|--------------|
340
- | Base | Core audio/vision/motion | `fastrtc`, `aiortc`, `openai`, `gradio`, `opencv-python`, `reachy_mini` |
341
- | `reachy_mini_wireless` | Wireless robot support | `PyGObject`, `gst-signalling` (GStreamer) |
342
- | `local_vision` | Local VLM processing | `torch`, `transformers`, `num2words` |
343
- | `yolo_vision` | YOLO head tracking | `ultralytics`, `supervision` |
344
- | `mediapipe_vision` | MediaPipe tracking | `mediapipe==0.10.14` |
345
- | `all_vision` | All vision features | Combination of above |
346
- | `dev` | Development tools | `pytest`, `ruff`, `mypy`, `pre-commit` |
347
-
348
- ## Common Troubleshooting
349
-
350
- **TimeoutError on startup**: Reachy Mini daemon not running. Install and start Reachy Mini SDK.
351
-
352
- **Vision not working**: Check that camera is connected and accessible. Use `--no-camera` to disable if not needed.
353
-
354
- **Wireless connection issues**: Ensure `--wireless-version` flag is used and daemon started with same flag. Requires `reachy_mini_wireless` extra.
355
-
356
- **Local vision slow**: SmolVLM2 benefits from GPU/MPS acceleration. Consider using default gpt-realtime vision if CPU-only.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
GEMINI.md ADDED
@@ -0,0 +1 @@
 
 
1
+ CLAUDE.md