deploy Gita Advisor as Gradio Space

- add app.py: Gradio ChatInterface wrapping the advisor pipeline
- add .gitignore: exclude .env, __pycache__, raw data, logs
- update README.md: add HF Spaces frontmatter
- update requirements.txt: add gradio>=4.0
- include artifacts/chroma/ (via LFS), optimized_advisor.json, corpus_enriched.jsonl

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.sqlite3 filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,14 @@
+.env
+__pycache__/
+*.pyc
+*.pyo
+data/raw/
+data/enrichment_cache.jsonl
+data/corpus.jsonl
+artifacts/gepa_logs/
+artifacts/gepa_state.bin
+artifacts/*.log
+Gita-advisor/
+sources_local/
+sources/
+.DS_Store

README.md

@@ -1,14 +1,125 @@
 ---
 title: Gita Advisor
-emoji: 📊
-colorFrom: indigo
-colorTo: green
+emoji: 🕉️
+colorFrom: yellow
+colorTo: orange
 sdk: gradio
-sdk_version: 6.13.0
+sdk_version: 5.33.0
 app_file: app.py
 pinned: false
-license: mit
-short_description: 'Bhagvad Gita Spiritual Advisor '
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Gītā Advisor
+
+A spiritual advisor grounded in Advaita Vedānta as taught by Śaṅkarācārya,
+optimized via DSPy + GEPA against a local LM Studio model. The advisor takes
+real-life questions or vents and produces responses that are empathetic to
+the felt experience, faithful to the non-dual lineage, and grounded in
+exact-cited verses from the Gītā with Śaṅkara's bhāṣya, the principal
+Upaniṣads, the Brahma Sūtras, and the prakaraṇa-granthas.
+
+## What makes this design unusual
+
+The first unusual choice is that the unit of retrieval is the verse, not
+the chunk. Scripture is not arbitrary prose: each Gītā śloka, each
+Upaniṣadic mantra, each sūtra is a sealed teaching unit with a stable
+citation reference. We index by `verse_id` (e.g. `bhagavad_gita_02_47`,
+which renders as `BG 2.47` in citations) so the advisor's references can be
+exact-match-verified against the retrieved set.
+
+The second unusual choice is that we use the local LLM, in a one-time
+offline pass, to enrich each verse with structured fields a real person's
+question can match against. A user does not write "I am experiencing
+rāga toward kāmya-karma"; they write "I worked on this for three years and
+it just failed." So we ask the local model, for each verse, to produce a
+plain-English paraphrase, the Vedāntic themes engaged, the life situations
+addressed, the emotions met, the practical teaching offered, and five
+hypothetical first-person questions the verse would speak to. We then
+embed three views of each verse — the literal translation, Śaṅkara's
+bhāṣya, and the LLM-enriched advisor view — and at retrieval time query
+all three and merge by verse ID.
+
+The advisor view dominates retrieval because that is where the language
+gap closes. The literal and bhāṣya views act as insurance against the
+enrichment pipeline missing a topic.
+
+## Where the texts come from
+
+Every source is unambiguously open. The verse-indexed JSON at
+`github.com/gita/gita`, released under the Unlicense, gives us Sanskrit
+plus IAST transliteration plus word-by-word glosses for the Gītā. Alladi
+Mahadeva Sastry's 1897 translation of Śaṅkara's Gītā Bhāṣya, in the public
+domain and full-text on archive.org, gives us Śaṅkara's commentary
+attached to each verse. The wisdomlib mirror of the *Sacred Books of the
+East* is staged for the Upaniṣad-with-Śaṅkara texts and the Brahma Sūtra
+bhāṣya; those parsers are registered but not yet implemented. See
+`sources_registry.py` for the complete catalog and `CLAUDE.md` for the
+licensing rationale.
+
+We deliberately exclude the modern Advaita Ashrama translations (active
+copyright), modern Ramaṇa and Nisargadatta editions, and Prabhupada's
+commentary. If you have your own license-cleared copies, drop them in
+`sources_local/` and the `plain_text` parser will fold them in.
+
+## Pipeline of commands
+
+```bash
+pip install -r requirements.txt
+
+# 1. Download the registered open sources to data/raw/<source_key>/
+python download_sources.py
+
+# 2. Parse + merge into data/corpus.jsonl (one verse per line)
+python ingest_corpus.py
+
+# 3. Enrich every verse via the local LLM. SLOW — overnight.
+#    Resumable; kill -9 is safe (append-mode cache).
+python enrich_corpus.py --limit 50    # smoke-test the prompt first
+python enrich_corpus.py               # then the real run
+
+# 4. Build the three-view Chroma index
+python knowledge_base.py --build
+
+# 5. Try a query against the index
+python knowledge_base.py --query "I just got laid off and feel hollow"
+
+# 6. Smoke-test the full advisor pipeline
+python smoke_test.py "I just got laid off and feel hollow"
+
+# 7. Generate the synthetic question dataset and run GEPA
+python dataset_generator.py --n 500
+python optimize_gepa.py --auto medium
+
+# 8. Open the chat CLI
+python chat.py
+```
+
+## Project structure
+
+The project is laid out so the data flow is left-to-right through the
+pipeline: each script reads what the previous one wrote, with all
+intermediate state on disk so any stage can be re-run independently. The
+data model lives in `corpus.py` (`Verse` and `EnrichedVerse` dataclasses)
+and is the contract between modules. The advisor itself is a `dspy.Module`
+that GEPA optimizes; the metric in `metrics.py` is the specification GEPA
+optimizes against, combining rule-based hygiene checks with an LLM-judge
+rubric and producing structured feedback for GEPA's reflection step. See
+`CLAUDE.md` for the full file map and the design commitments that should
+not be silently broken.
+
+## Configuration
+
+`config.py` reads a small number of environment variables. The two that
+matter most are `LM_STUDIO_BASE` (defaults to `http://localhost:1234/v1`)
+and `LOCAL_MODEL` (defaults to `google/gemma-4-26b-a4b`, but copy whatever
+LM Studio reports verbatim). The embedding model defaults to BGE-small on
+Apple Silicon's MPS device; switch `EMBED_DEVICE` to `cpu` if you are not
+on Apple Silicon.
+
+## License
+
+The code in this repository is yours to use. The texts in `data/raw/` come
+with their own licenses, all unambiguously open and tracked in
+`sources_registry.py`. Attributions for translators are preserved through
+the pipeline and surfaced in citation footers.
+# gita_advisor

advisor.py

@@ -0,0 +1,157 @@
+"""
+advisor.py — the composed DSPy module.
+
+This is what GEPA optimizes. It chains four predictors:
+
+    UnderstandQuery   →  PlanRetrieval  →  [retrieve] → SelectPassages → SynthesizeAdvice
+
+Each predictor uses ChainOfThought so GEPA has a `reasoning` field to inspect
+in its reflection step. The retriever itself is not optimized (it's vector
+search), but the *queries given to it* are — that's where PlanRetrieval lives.
+"""
+
+from __future__ import annotations
+import json
+from dataclasses import dataclass
+from typing import Any
+import dspy
+
+from signatures import (
+    UnderstandQuery,
+    PlanRetrieval,
+    SelectPassages,
+    SynthesizeAdvice,
+)
+from knowledge_base import AdvaitaRetriever, format_passages_for_llm
+import config
+
+
+@dataclass
+class AdviceTrace:
+    """Everything the pipeline produced — useful for the metric to reason over."""
+    user_question: str
+    felt_emotion: str
+    surface_concern: str
+    deeper_concern: str
+    vedantic_themes: list[str]
+    queries: list[str]
+    retrieved_passages: list[dict]    # raw hits with metadata
+    selected_indices: list[int]
+    selection_rationale: str
+    response: str
+    sources_cited: list[str]
+
+
+class GitaAdvisor(dspy.Module):
+    def __init__(self, retriever: AdvaitaRetriever | None = None):
+        super().__init__()
+        self.understand = dspy.ChainOfThought(UnderstandQuery)
+        self.plan = dspy.ChainOfThought(PlanRetrieval)
+        self.select = dspy.ChainOfThought(SelectPassages)
+        self.synthesize = dspy.ChainOfThought(SynthesizeAdvice)
+        # Retriever is not a Predictor; held as a plain attribute so DSPy
+        # introspection ignores it during optimization.
+        self._retriever = retriever or AdvaitaRetriever()
+
+    def forward(
+        self,
+        user_question: str,
+        history: dspy.History | None = None,
+        _stage_cb=None,
+    ) -> dspy.Prediction:
+        if history is None:
+            history = dspy.History(messages=[])
+
+        # 1. Understand — history lets it interpret follow-ups correctly
+        if _stage_cb:
+            _stage_cb("understanding your question...")
+        u = self.understand(
+            history=history,
+            user_question=user_question,
+        )
+
+        # 2. Plan retrieval queries
+        if _stage_cb:
+            _stage_cb("planning search queries...")
+        p = self.plan(
+            surface_concern=u.surface_concern,
+            deeper_concern=u.deeper_concern,
+            vedantic_themes=u.vedantic_themes,
+        )
+        queries = p.queries[: config.N_RETRIEVAL_QUERIES] if p.queries else [u.deeper_concern]
+
+        # 3. Retrieve
+        if _stage_cb:
+            _stage_cb("searching scriptures...")
+        hits = self._retriever.search_many(queries, k_per=config.TOP_K_RETRIEVE)
+        # Cap candidate set so the selector prompt stays focused
+        candidates = hits[: max(8, config.TOP_K_RETRIEVE)]
+        candidates_text = format_passages_for_llm(candidates)
+        # Pre-serialize hits to dicts so the dspy.Prediction we return below
+        # can be pickled by GEPA's bookkeeping. The metric reads from these
+        # dicts, not from Hit objects, so it doesn't need the knowledge_base
+        # import either.
+        candidates_as_dicts = [h.to_dict() for h in candidates]
+
+        # 4. Select — tell the selector what's already been cited so it prefers fresh sources
+        if _stage_cb:
+            _stage_cb("selecting passages...")
+        previously_cited = [
+            src
+            for msg in history.messages
+            for src in msg.get("sources_cited", [])
+        ]
+        s = self.select(
+            deeper_concern=u.deeper_concern,
+            candidate_passages=candidates_text,
+            previously_cited=previously_cited,
+        )
+        # Defensive: clamp indices to valid range
+        valid_idx = [
+            i for i in (s.selected_indices or [])
+            if isinstance(i, int) and 1 <= i <= len(candidates)
+        ]
+        if not valid_idx:
+            # Fallback: take the top-3 candidates so synthesis isn't starved
+            valid_idx = list(range(1, min(4, len(candidates) + 1)))
+
+        selected = [candidates[i - 1] for i in valid_idx]
+        selected_text = format_passages_for_llm(selected)
+
+        # 5. Synthesize — history lets it build across turns, avoid repetition
+        if _stage_cb:
+            _stage_cb("composing response...")
+        a = self.synthesize(
+            history=history,
+            user_question=user_question,
+            felt_emotion=u.felt_emotion,
+            deeper_concern=u.deeper_concern,
+            selected_passages=selected_text,
+        )
+
+        return dspy.Prediction(
+            response=a.response,
+            sources_cited=a.sources_cited or [],
+            synthesis_reasoning=getattr(a, "reasoning", ""),
+            # Carry intermediate state for the metric / debugging:
+            felt_emotion=u.felt_emotion,
+            surface_concern=u.surface_concern,
+            deeper_concern=u.deeper_concern,
+            vedantic_themes=u.vedantic_themes,
+            queries=queries,
+            retrieved_passages=candidates_as_dicts,
+            selected_indices=valid_idx,
+            selection_rationale=s.selection_rationale,
+        )
+
+
+def load_optimized(path: str | None = None) -> GitaAdvisor:
+    """Load an advisor with GEPA-optimized prompts if available, else fresh."""
+    advisor = GitaAdvisor()
+    p = path or str(config.OPTIMIZED_PROGRAM_PATH)
+    try:
+        advisor.load(p)
+        print(f"Loaded optimized advisor from {p}")
+    except FileNotFoundError:
+        print(f"No optimized program at {p} — using base prompts.")
+    return advisor

app.py

@@ -0,0 +1,72 @@
+"""
+app.py — Gradio web interface for the Gītā Advisor.
+
+Wraps the same advisor pipeline as chat.py but exposes it as a Gradio
+ChatInterface suitable for Hugging Face Spaces (free CPU tier).
+
+Deploy:
+  - Set GEMINI_API_KEY as a Space Secret (Space Settings → Secrets)
+  - Push this file + all project files + artifacts/chroma/ + data/corpus_enriched.jsonl
+"""
+
+import gradio as gr
+import dspy
+
+import config
+from advisor import load_optimized
+from knowledge_base import AdvaitaRetriever
+
+# ── startup — runs once when the Space boots ───────────────────────────────────
+config.configure_dspy()
+_advisor = load_optimized()
+
+# Pre-warm retriever so the first user request isn't slow
+_retriever = AdvaitaRetriever()
+_retriever._ensure()
+
+
+# ── chat handler ───────────────────────────────────────────────────────────────
+def chat(message: str, history: list) -> str:
+    # Gradio type="messages" passes history as list of {"role": ..., "content": ...}
+    dspy_msgs = []
+    i = 0
+    while i + 1 < len(history):
+        user_msg = history[i]
+        bot_msg = history[i + 1]
+        if user_msg.get("role") == "user" and bot_msg.get("role") == "assistant":
+            dspy_msgs.append({
+                "user_question": user_msg["content"],
+                "response": bot_msg["content"],
+                "sources_cited": [],
+            })
+        i += 2
+    dspy_history = dspy.History(messages=dspy_msgs)
+
+    pred = _advisor(user_question=message, history=dspy_history)
+
+    reply = pred.response
+    if pred.sources_cited:
+        reply += "\n\n---\n**Sources:** " + " · ".join(pred.sources_cited)
+    return reply
+
+
+# ── Gradio app ─────────────────────────────────────────────────────────────────
+demo = gr.ChatInterface(
+    fn=chat,
+    title="Gītā Advisor",
+    description=(
+        "A spiritual advisor grounded in Advaita Vedānta as taught by Śaṅkarācārya. "
+        "Speak from where you actually are. The advisor cites exact verses from the "
+        "Gītā with Śaṅkara's commentary."
+    ),
+    type="messages",
+    examples=[
+        "I just got laid off and feel like nothing makes sense.",
+        "I'm terrified of dying. Is that irrational?",
+        "I keep hurting the people I love without meaning to.",
+        "I've been meditating for years but still feel empty. What am I missing?",
+    ],
+)
+
+if __name__ == "__main__":
+    demo.launch()

artifacts/chroma/1cb22ce3-5b5d-4ea7-84da-8e74f131266a/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e83b9e9a3e97ce6dc11715a48ebb2e9bc6f5b24f91c82d3fbc8d6b46085afb44
+size 1174876

artifacts/chroma/1cb22ce3-5b5d-4ea7-84da-8e74f131266a/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90261dc9d8649c19b95dcba5fa4ef3bb2f5a64b26da0aefc9e58ba7a3f2fcbe0
+size 100

artifacts/chroma/1cb22ce3-5b5d-4ea7-84da-8e74f131266a/index_metadata.pickle

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b626b118ac6ec87f4d2ecdb13193aa4eaadc0514fe351c284c77edcc0b491fea
+size 40786

artifacts/chroma/1cb22ce3-5b5d-4ea7-84da-8e74f131266a/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b57dc524397d289a5ec9f2dfd69bf88c097d397d354c600fff6cbd958cadda89
+size 2804

artifacts/chroma/1cb22ce3-5b5d-4ea7-84da-8e74f131266a/link_lists.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdb3c70e56609aecebbde493e89cc3f2eff23d6375c762bca900dd5edc46738b
+size 6204

artifacts/chroma/1f1f3474-209f-41fb-91d3-d45fd026fb05/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66cd81ca458bd620e24b378f6ce96e6d77ba4c8789d4ece914775c339be10e26
+size 167600

artifacts/chroma/1f1f3474-209f-41fb-91d3-d45fd026fb05/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0e81c3b22454233bc12d0762f06dcca48261a75231cf87c79b75e69a6c00150
+size 100

artifacts/chroma/1f1f3474-209f-41fb-91d3-d45fd026fb05/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
+size 400

artifacts/chroma/52cdeb15-0631-44ed-8618-782f1d4d27bb/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:201b998f2a013f78cea5960b05174ceffedbd046c4dfc10a8d2492ff8a1398a7
+size 167600

artifacts/chroma/52cdeb15-0631-44ed-8618-782f1d4d27bb/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0e81c3b22454233bc12d0762f06dcca48261a75231cf87c79b75e69a6c00150
+size 100

artifacts/chroma/52cdeb15-0631-44ed-8618-782f1d4d27bb/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
+size 400

artifacts/chroma/8707047c-50b3-41ba-ad04-329e93917e30/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:121a03535f783813fa1ec964dd84095d2da0d83c0e5dad98955bdf0e252b33d8
+size 167600

artifacts/chroma/8707047c-50b3-41ba-ad04-329e93917e30/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0e81c3b22454233bc12d0762f06dcca48261a75231cf87c79b75e69a6c00150
+size 100

artifacts/chroma/8707047c-50b3-41ba-ad04-329e93917e30/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
+size 400

artifacts/chroma/9c71c1cd-5694-4e9f-abe6-12ba1e74225b/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68a485ecde933131e39b14c078dc82c3e0e27454cdb837d76f4be92f00bef026
+size 1136328

artifacts/chroma/9c71c1cd-5694-4e9f-abe6-12ba1e74225b/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ce56f617adc30c61cd845ab6c84720756b445d622b4bc3160ce70dc5ce91e7e
+size 100

artifacts/chroma/9c71c1cd-5694-4e9f-abe6-12ba1e74225b/index_metadata.pickle

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bde285e65f8c89d9d82ad8cd02cd29c9bfcbe7009e37a2ef50f2fc58dd630af1
+size 39452

artifacts/chroma/9c71c1cd-5694-4e9f-abe6-12ba1e74225b/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:360e5acdc3007feb7ef856ed67807c4a8995beb723091091828f729ccba9dbcb
+size 2712

artifacts/chroma/9c71c1cd-5694-4e9f-abe6-12ba1e74225b/link_lists.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fef3ffac7707ea201b174e47e15c161162d3f38b6fdbeccc43aab1af3c35e617
+size 6044

artifacts/chroma/chroma.sqlite3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acd63ba75ad75f6234302822709e44e8e560d80e120f31519c20f73d41821d22
+size 20459520

artifacts/chroma/d091e62b-3e8b-4cd6-842e-ff3411b384f5/data_level0.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88b9416504c97719067232f4ef2a042eb4015b033f8c4a3137cd90e9f2faa468
+size 167600

artifacts/chroma/d091e62b-3e8b-4cd6-842e-ff3411b384f5/header.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0e81c3b22454233bc12d0762f06dcca48261a75231cf87c79b75e69a6c00150
+size 100

artifacts/chroma/d091e62b-3e8b-4cd6-842e-ff3411b384f5/length.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12e561363385e9dfeeab326368731c030ed4b374e7f5897ac819159d2884c5
+size 400

artifacts/optimized_advisor.json

@@ -0,0 +1,157 @@
+{
+  "understand.predict": {
+    "traces": [],
+    "train": [],
+    "demos": [],
+    "signature": {
+      "instructions": "Read the user's life situation carefully, taking into account the full\nconversation so far. If there is prior exchange, use it to understand\nfollow-up messages, references like 'what you said earlier', or shifts in\nthe user's emotional state across turns. Identify the felt emotion, the\nunderlying spiritual concern (not just the surface complaint), and the\nVedāntic themes that are most relevant — drawing only from concepts native\nto Advaita Vedānta.",
+      "fields": [
+        {
+          "prefix": "History:",
+          "description": "Prior turns as a list of message dicts with 'user_question' and 'response' keys. Empty history means this is the first message."
+        },
+        {
+          "prefix": "User Question:",
+          "description": "The user's current message; may be a question, a vent, a follow-up, or a description of a situation."
+        },
+        {
+          "prefix": "Reasoning:",
+          "description": "${reasoning}"
+        },
+        {
+          "prefix": "Felt Emotion:",
+          "description": "The dominant emotion the user is experiencing, named precisely (e.g. 'anticipatory grief', not just 'sad')."
+        },
+        {
+          "prefix": "Surface Concern:",
+          "description": "What the user is literally asking about, in one sentence."
+        },
+        {
+          "prefix": "Deeper Concern:",
+          "description": "The underlying existential/spiritual concern — usually about identity, attachment, fear, dharma, or meaning — that the surface concern is a symptom of. One sentence."
+        },
+        {
+          "prefix": "Vedantic Themes:",
+          "description": "2-4 Advaita-Vedānta concepts most relevant to this situation. Use Sanskrit terms with brief gloss, e.g. 'adhyāsa (superimposition of self onto roles)', 'vairāgya (dispassion)', 'sākṣī (witness consciousness)'."
+        }
+      ]
+    },
+    "lm": null
+  },
+  "plan.predict": {
+    "traces": [],
+    "train": [],
+    "demos": [],
+    "signature": {
+      "instructions": "Given the user's situation and identified themes, generate diverse search\nqueries to find relevant passages from the Advaita corpus (Bhagavad Gītā with\nŚaṅkara bhāṣya, Upaniṣads, Brahma Sūtras, prakaraṇa texts). Each query should\ntarget a different angle — one query about the philosophical principle,\none about a parallel situation in the texts, one about the practical\nteaching offered by the lineage.",
+      "fields": [
+        {
+          "prefix": "Surface Concern:",
+          "description": "${surface_concern}"
+        },
+        {
+          "prefix": "Deeper Concern:",
+          "description": "${deeper_concern}"
+        },
+        {
+          "prefix": "Vedantic Themes:",
+          "description": "${vedantic_themes}"
+        },
+        {
+          "prefix": "Reasoning:",
+          "description": "${reasoning}"
+        },
+        {
+          "prefix": "Queries:",
+          "description": "3 distinct search queries (each 5-15 words). Vary in angle: principle, parallel, practice."
+        }
+      ]
+    },
+    "lm": null
+  },
+  "select.predict": {
+    "traces": [],
+    "train": [],
+    "demos": [],
+    "signature": {
+      "instructions": "From the retrieved candidate passages, select the ones that genuinely\nspeak to this user's situation. Prefer primary sources (Gītā verses,\nUpaniṣadic mantras, Śaṅkara's bhāṣya) over secondary or modern commentary\nwhen both are available. Reject passages that are merely topically adjacent\nbut don't address the actual spiritual concern. Avoid re-selecting passages\nwhose source was already cited in a prior turn — prefer fresh ground.",
+      "fields": [
+        {
+          "prefix": "Deeper Concern:",
+          "description": "${deeper_concern}"
+        },
+        {
+          "prefix": "Candidate Passages:",
+          "description": "Numbered candidate passages with source attribution."
+        },
+        {
+          "prefix": "Previously Cited:",
+          "description": "Source references already cited in earlier turns of this conversation (e.g. ['BG 2.47', 'BG 18.66']). Prefer passages not on this list so the conversation covers new ground. Empty list on the first turn."
+        },
+        {
+          "prefix": "Reasoning:",
+          "description": "${reasoning}"
+        },
+        {
+          "prefix": "Selected Indices:",
+          "description": "Indices (1-based) of the 2-4 most relevant passages."
+        },
+        {
+          "prefix": "Selection Rationale:",
+          "description": "One sentence per selection explaining why that passage speaks to this concern."
+        }
+      ]
+    },
+    "lm": null
+  },
+  "synthesize.predict": {
+    "traces": [],
+    "train": [],
+    "demos": [],
+    "signature": {
+      "instructions": "Compose a response that is grounded in Advaita Vedānta as taught by\nŚaṅkarācārya, empathetic to the user's felt experience, and practically\nuseful for their situation. Honor the two-truths distinction: meet the user\nin vyāvahārika (transactional reality) without ever denying the\npāramārthika (absolute) view. Cite specific verses/passages by reference,\nintegrate them into prose rather than dumping quotes, and keep wit gentle —\nlight around the cosmic predicament, never light about the user's pain.\n\nIf history has prior turns: do not repeat citations or teachings already\ngiven; build on or deepen what was said; acknowledge any shift the user has\nexpressed since the last turn. If the user is following up, open by briefly\nacknowledging the continuity before moving forward.",
+      "fields": [
+        {
+          "prefix": "History:",
+          "description": "Prior turns as a list of message dicts with 'user_question' and 'response' keys. Use this to avoid repetition and to build across turns."
+        },
+        {
+          "prefix": "User Question:",
+          "description": "${user_question}"
+        },
+        {
+          "prefix": "Felt Emotion:",
+          "description": "${felt_emotion}"
+        },
+        {
+          "prefix": "Deeper Concern:",
+          "description": "${deeper_concern}"
+        },
+        {
+          "prefix": "Selected Passages:",
+          "description": "The selected passages with full source attribution."
+        },
+        {
+          "prefix": "Reasoning:",
+          "description": "${reasoning}"
+        },
+        {
+          "prefix": "Response:",
+          "description": "The advisor's reply to the user. 250-450 words. Open by acknowledging the felt experience. Move into the Vedāntic perspective. Cite at least one primary source (Gītā chapter:verse, Upaniṣad name + section, etc.). Close with a concrete practice or shift in perspective they can try this week. Address the user as 'you' throughout. Avoid Western therapy clichés."
+        },
+        {
+          "prefix": "Sources Cited:",
+          "description": "Source references actually cited in the response, e.g. 'BG 2.47', 'Bṛhadāraṇyaka Up. 4.4.5', 'Vivekacūḍāmaṇi 11'."
+        }
+      ]
+    },
+    "lm": null
+  },
+  "metadata": {
+    "dependency_versions": {
+      "python": "3.11",
+      "dspy": "3.2.0",
+      "cloudpickle": "3.1"
+    }
+  }
+}

chat.py

@@ -0,0 +1,392 @@
+"""
+chat.py — interactive conversation with the advisor.
+
+By default it loads the GEPA-optimized program from artifacts/. If that file
+doesn't exist yet, it falls back to the un-optimized base prompts so you can
+sanity-check the pipeline before running optimization.
+
+Flags:
+  --debug       Show intermediate pipeline state (felt emotion, queries, etc.)
+  --thinking    Show the full synthesis reasoning trace (default: first 6 lines)
+  --no-thinking Hide the reasoning trace entirely
+
+After each response, source references are printed with numbers.
+  show <N|ref>    Display the verse text, translation, and Śaṅkara's bhāṣya.
+  explain <N|ref> Show the verse then stream a contextual explanation of how
+                  it applies to the current conversation.
+"""
+
+from __future__ import annotations
+import argparse
+import time
+import threading
+from typing import Optional
+
+import dspy
+from rich.console import Console
+from rich.live import Live
+from rich.markdown import Markdown
+from rich.panel import Panel
+from rich.rule import Rule
+from rich.text import Text
+
+import config
+from advisor import load_optimized
+from corpus import EnrichedVerse, Verse, read_jsonl_enriched, read_jsonl_verses
+
+
+# ── speed constants ────────────────────────────────────────────────────────────
+_THINKING_CPS = 800   # chars/sec for reasoning stream (secondary content, fast)
+_RESPONSE_CPS = 300   # chars/sec for advisor response (primary content)
+_THINKING_PREVIEW = 6 # lines shown in collapsed thinking mode
+
+
+# ── verse corpus lookup ────────────────────────────────────────────────────────
+def _load_verse_lookup() -> dict[str, Verse]:
+    """Build a case-insensitive verse_ref → Verse dict from the corpus."""
+    lookup: dict[str, Verse] = {}
+    enriched = config.DATA_DIR / "corpus_enriched.jsonl"
+    plain = config.DATA_DIR / "corpus.jsonl"
+
+    if enriched.exists():
+        loader, path = read_jsonl_enriched, enriched
+    elif plain.exists():
+        loader, path = read_jsonl_verses, plain
+    else:
+        return lookup
+
+    for verse in loader(path):
+        lookup[verse.verse_ref.lower().strip()] = verse
+    return lookup
+
+
+def _find_verse(lookup: dict, ref: str) -> Optional[Verse]:
+    return lookup.get(ref.lower().strip())
+
+
+def _resolve_ref(arg: str, sources_cited: list[str]) -> str:
+    """Turn '1' → sources_cited[0], or return arg unchanged for direct ref lookup."""
+    try:
+        n = int(arg.strip())
+        if 1 <= n <= len(sources_cited):
+            return sources_cited[n - 1]
+    except ValueError:
+        pass
+    return arg.strip()
+
+
+# ── DSPy signature for contextual explanation ─────────────────────────────────
+class _ExplainInContext(dspy.Signature):
+    """You are the Gītā Advisor continuing a conversation. The user has asked
+    you to unpack a specific verse or passage you cited. Explain what it means
+    and why it speaks precisely to their situation — go deeper than the initial
+    response did. Reference the user's words. Close with one concrete way to
+    hold or work with this text this week."""
+
+    verse_ref: str = dspy.InputField()
+    verse_content: str = dspy.InputField(
+        desc="Translation, original text (if available), and Śaṅkara's commentary."
+    )
+    conversation_context: str = dspy.InputField(
+        desc="The user's question and the advisor's response where this verse was cited."
+    )
+
+    explanation: str = dspy.OutputField(
+        desc="150-250 words. Grounded in Advaita. Do not merely restate the translation. "
+             "End with a practical suggestion for this week."
+    )
+
+
+# ── streaming helpers ─────────────────────────────────────────────────────────
+def _stream_chars(console: Console, text: str, cps: int):
+    """Write text to the terminal character by character."""
+    if not text:
+        return
+    delay = 1.0 / cps
+    for ch in text:
+        console.file.write(ch)
+        console.file.flush()
+        time.sleep(delay)
+    console.file.write("\n")
+    console.file.flush()
+
+
+def _stream_response(console: Console, text: str, cps: int = _RESPONSE_CPS):
+    """Stream the advisor response into a growing Markdown Panel via Rich Live."""
+    if not text:
+        return
+    displayed = ""
+    delay = 1.0 / cps
+    with Live(console=console, refresh_per_second=min(cps, 30)) as live:
+        for ch in text:
+            displayed += ch
+            live.update(Panel(
+                Markdown(displayed),
+                title="[bold]advisor[/bold]",
+                border_style="yellow",
+                padding=(1, 2),
+            ))
+            time.sleep(delay)
+
+
+def _show_thinking(console: Console, reasoning: str, full: bool):
+    """Stream the synthesis reasoning below a dim rule, collapsed to _THINKING_PREVIEW lines."""
+    if not reasoning:
+        return
+
+    lines = reasoning.strip().splitlines()
+    if not full and len(lines) > _THINKING_PREVIEW:
+        display = "\n".join(lines[:_THINKING_PREVIEW])
+        n_hidden = len(lines) - _THINKING_PREVIEW
+    else:
+        display = "\n".join(lines)
+        n_hidden = 0
+
+    console.print(Rule("[dim]thinking[/dim]", style="dim blue"))
+    # Write dim italic via ANSI since we're streaming to file directly
+    # (Rich markup can't be applied char-by-char; dim is cosmetic here)
+    _stream_chars(console, display, cps=_THINKING_CPS)
+
+    if n_hidden:
+        console.print(f"[dim]  ↳ {n_hidden} more lines — use --thinking to expand[/dim]")
+    console.print()
+
+
+# ── verse display helpers ─────────────────────────────────────────────────────
+def _show_verse(console: Console, verse: Verse):
+    """Render a verse with its translation, original text, and commentary."""
+    body = Text()
+
+    if verse.sanskrit:
+        body.append(verse.sanskrit + "\n", style="bold")
+    if verse.transliteration:
+        body.append(verse.transliteration + "\n", style="italic dim")
+
+    if verse.translation:
+        label = f"Translation ({verse.translator})" if verse.translator else "Translation"
+        body.append(f"\n{label}:\n", style="dim")
+        body.append(verse.translation + "\n")
+
+    if verse.bhashya:
+        translator_note = f" ({verse.bhashya_translator})" if verse.bhashya_translator else ""
+        body.append(f"\nŚaṅkara's Bhāṣya{translator_note}:\n", style="dim")
+        preview = verse.bhashya[:800] + ("…" if len(verse.bhashya) > 800 else "")
+        body.append(preview + "\n", style="dim")
+
+    ev = verse if isinstance(verse, EnrichedVerse) else None
+    if ev and ev.paraphrase:
+        body.append("\nTeaching: ", style="bold dim")
+        body.append(ev.paraphrase + "\n", style="dim")
+    if ev and ev.themes:
+        body.append("Themes: ", style="bold dim")
+        body.append(", ".join(ev.themes) + "\n", style="dim")
+    if ev and ev.practical_teaching:
+        body.append("Practical shift: ", style="bold dim")
+        body.append(ev.practical_teaching + "\n", style="dim")
+
+    section = verse.section_display or verse.section
+    subtitle = verse.work_display + (f" — {section}" if section else "")
+    console.print(Panel(
+        body,
+        title=f"[bold]{verse.verse_ref}[/bold]",
+        subtitle=f"[dim]{subtitle}[/dim]",
+        border_style="cyan",
+        padding=(1, 2),
+    ))
+
+
+def _explain_in_context(
+    console: Console,
+    verse: Verse,
+    history_messages: list[dict],
+    cps: int = _RESPONSE_CPS,
+):
+    """Call the LM to explain the verse in context of the last conversation turn."""
+    if history_messages:
+        last = history_messages[-1]
+        context = (
+            f"User: {last.get('user_question', '')}\n\n"
+            f"Advisor: {last.get('response', '')}"
+        )
+    else:
+        context = "No prior conversation."
+
+    bits = []
+    if verse.translation:
+        bits.append(f"Translation: {verse.translation}")
+    if verse.sanskrit:
+        bits.append(f"Sanskrit: {verse.sanskrit}")
+    if verse.bhashya:
+        bits.append(f"Śaṅkara's commentary: {verse.bhashya[:600]}")
+    ev = verse if isinstance(verse, EnrichedVerse) else None
+    if ev and ev.paraphrase:
+        bits.append(f"Teaching: {ev.paraphrase}")
+    verse_content = "\n\n".join(bits)
+
+    explainer = dspy.ChainOfThought(_ExplainInContext)
+    with console.status("[dim]expanding...[/dim]", spinner="dots"):
+        try:
+            result = explainer(
+                verse_ref=verse.verse_ref,
+                verse_content=verse_content,
+                conversation_context=context,
+            )
+            explanation = result.explanation
+        except Exception as exc:
+            console.print(f"[red]Could not generate explanation: {exc}[/red]")
+            return
+
+    console.print()
+    _stream_response(console, explanation, cps=cps)
+
+
+# ── main loop ─────────────────────────────────────────────────────────────────
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--program", default=str(config.OPTIMIZED_PROGRAM_PATH))
+    ap.add_argument("--debug", action="store_true",
+                    help="Show intermediate pipeline state for each turn.")
+    ap.add_argument("--thinking", action="store_true",
+                    help="Show full synthesis reasoning trace (default: first 6 lines).")
+    ap.add_argument("--no-thinking", action="store_true", dest="no_thinking",
+                    help="Hide the reasoning trace entirely.")
+    args = ap.parse_args()
+
+    config.configure_dspy()
+    advisor = load_optimized(args.program)
+    console = Console()
+
+    with console.status("[dim]loading corpus...[/dim]", spinner="dots"):
+        verse_lookup = _load_verse_lookup()
+
+    console.print(Panel.fit(
+        "[bold]Gītā Advisor[/bold]\n\n"
+        "Speak from where you actually are.\n"
+        "After a response: [italic]show <N>[/italic] to read a cited verse · "
+        "[italic]explain <N>[/italic] for contextual breakdown.\n"
+        "Type [italic]exit[/italic] or Ctrl-D to leave.",
+        border_style="cyan",
+    ))
+
+    history = dspy.History(messages=[])
+    last_pred = None
+
+    while True:
+        try:
+            console.print()
+            console.print("[bold cyan]you:[/bold cyan] ", end="")
+            line = input().strip()
+        except (EOFError, KeyboardInterrupt):
+            console.print("\n[dim]नमस्ते।[/dim]")
+            return
+
+        if not line:
+            continue
+        if line.lower() in {"exit", "quit", ":q"}:
+            console.print("[dim]नमस्ते।[/dim]")
+            return
+
+        # ── source exploration commands ───────────────────────────────────────
+        cmd_lower = line.lower()
+        if cmd_lower.startswith(("show ", "explain ")):
+            if last_pred is None:
+                console.print("[dim]No sources yet — ask a question first.[/dim]")
+                continue
+            cmd, _, arg = line.partition(" ")
+            ref = _resolve_ref(arg, last_pred.sources_cited)
+            verse = _find_verse(verse_lookup, ref)
+            if verse is None:
+                console.print(f"[dim]'{ref}' not found in corpus.[/dim]")
+                if last_pred.sources_cited:
+                    hint = "  ".join(
+                        f"[{i+1}] {r}" for i, r in enumerate(last_pred.sources_cited)
+                    )
+                    console.print(f"[dim]Available: {hint}[/dim]")
+                continue
+            _show_verse(console, verse)
+            if cmd.lower() == "explain":
+                _explain_in_context(console, verse, history.messages)
+            continue
+
+        # ── normal question — run pipeline in background with live stage progress ──
+        pred = None
+        error = None
+        stage = ["initializing..."]
+        done = threading.Event()
+
+        def run_advisor():
+            nonlocal pred, error
+            try:
+                pred = advisor(
+                    user_question=line,
+                    history=history,
+                    _stage_cb=lambda msg: stage.__setitem__(0, msg),
+                )
+            except Exception as exc:
+                error = exc
+            finally:
+                done.set()
+
+        threading.Thread(target=run_advisor, daemon=True).start()
+
+        with Live(console=console, refresh_per_second=8) as live:
+            while not done.wait(timeout=0.12):
+                live.update(Text(f"  ◌  {stage[0]}", style="dim"))
+            live.update(Text(""))
+
+        if error:
+            console.print(f"[red]Error: {error}[/red]")
+            continue
+
+        last_pred = pred
+        history.messages.append({
+            "user_question": line,
+            "response": pred.response,
+            "sources_cited": pred.sources_cited,
+        })
+
+        # debug trace
+        if args.debug:
+            console.print(Rule("[dim]debug[/dim]", style="dim"))
+            console.print(f"[dim]felt:[/dim]    {pred.felt_emotion}")
+            console.print(f"[dim]surface:[/dim] {pred.surface_concern}")
+            console.print(f"[dim]deeper:[/dim]  {pred.deeper_concern}")
+            console.print(f"[dim]themes:[/dim]  {', '.join(pred.vedantic_themes)}")
+            console.print(f"[dim]queries:[/dim] {pred.queries}")
+            console.print(f"[dim]selected:[/dim] {pred.selected_indices}")
+            for i in pred.selected_indices:
+                if 1 <= i <= len(pred.retrieved_passages):
+                    h = pred.retrieved_passages[i - 1]
+                    m = h["meta"]
+                    console.print(
+                        f"  [dim]→ [{m['tier']}] {m['work']}"
+                        f"{' — ' + m['section'] if m.get('section') else ''}"
+                        f"  (score {h['score']:.3f})[/dim]"
+                    )
+            console.print(Rule(style="dim"))
+
+        # thinking section
+        if not args.no_thinking:
+            _show_thinking(
+                console,
+                getattr(pred, "synthesis_reasoning", ""),
+                full=args.thinking,
+            )
+
+        # stream the response
+        console.print()
+        _stream_response(console, pred.response)
+
+        # source footer with hints
+        if pred.sources_cited:
+            numbered = "  ".join(
+                f"[{i+1}] {r}" for i, r in enumerate(pred.sources_cited)
+            )
+            console.print(f"\n[dim]sources: {numbered}[/dim]")
+            console.print(
+                "[dim]  → show <N> to read the verse  ·  explain <N> for contextual breakdown[/dim]"
+            )
+
+
+if __name__ == "__main__":
+    main()

config.py

@@ -0,0 +1,201 @@
+"""
+config.py — central configuration for the Gītā Advisor.
+
+Three LMs are configured:
+
+  - TASK_LM:       the local model running in LM Studio. Used at inference
+                   time (understanding, retrieval planning, advice synthesis).
+
+  - ENRICH_LM:     Claude Sonnet (API) for the offline enrichment pass.
+                   The local 26B model truncates structured output at 1500
+                   tokens and drops fields. Claude handles all six fields
+                   cleanly in one call and costs ~$12-15 for the full 701-
+                   verse corpus (one-time). Set ANTHROPIC_API_KEY in env.
+
+  - REFLECTION_LM: gpt-4o (OpenAI) for GEPA's reflection step.
+                   GEPA asks the reflection LM to read metric feedback and
+                   propose rewritten prompts — this scales strongly with
+                   model quality. gpt-4o reasons well enough to handle
+                   nuanced Advaita feedback without breaking the budget.
+                   Same OPENAI_API_KEY as enrichment.
+"""
+
+from __future__ import annotations
+import os
+import re
+from pathlib import Path
+import dspy
+import dspy.adapters.chat_adapter as _chat_adapter_module
+from dotenv import load_dotenv
+
+# Gemma (and some other local models) output `[[ ## field ]]` without the closing `##`
+# that DSPy's ChatAdapter expects (`[[ ## field ## ]]`). Patch the module-level regex
+# to accept both forms before any adapter is instantiated.
+_chat_adapter_module.field_header_pattern = re.compile(r"\[\[ ## (\w+)(?:\s*##)? \]\]")
+
+load_dotenv(Path(__file__).parent / ".env")  # explicit path; works from any cwd
+
+# ──────────────────────────── Paths ────────────────────────────
+ROOT = Path(__file__).parent.resolve()
+SOURCES_DIR = ROOT / "sources"
+DATA_DIR = ROOT / "data"
+ARTIFACTS_DIR = ROOT / "artifacts"
+CHROMA_DIR = ARTIFACTS_DIR / "chroma"
+
+for d in (SOURCES_DIR, DATA_DIR, ARTIFACTS_DIR, CHROMA_DIR):
+    d.mkdir(parents=True, exist_ok=True)
+
+DATASET_PATH = DATA_DIR / "synthetic_questions.jsonl"
+OPTIMIZED_PROGRAM_PATH = ARTIFACTS_DIR / "optimized_advisor.json"
+
+# ──────────────────────────── Task LM — Gemini API (preferred) ───────────────────────────
+# When GEMINI_API_KEY is set, route the task LM through Google AI Studio.
+# Same Gemma 4 26B weights, but no local GPU required and the free tier is
+# sufficient for inference + GEPA optimization runs.
+GEMINI_API_KEY = os.getenv("GEMINI_API_KEY", "")
+GEMINI_TASK_MODEL = os.getenv("GEMINI_TASK_MODEL", "gemini/gemma-4-26b-a4b-it")
+
+GEMINI_TASK_LM_KWARGS = dict(
+    api_key=GEMINI_API_KEY,
+    temperature=0.6,
+    # Gemma 4 thinking tokens count against max_tokens in the Gemini API.
+    # Each pipeline call burns ~3-4k reasoning tokens before writing output,
+    # so 4096 gets truncated. 16384 gives comfortable headroom for both.
+    max_tokens=16384,
+    cache=True,
+)
+
+# ──────────────────────────── Task LM — LM Studio fallback ───────────────────────────────
+LM_STUDIO_BASE = os.getenv("LM_STUDIO_BASE", "http://localhost:1234/v1")
+LOCAL_MODEL = os.getenv("LOCAL_MODEL", "google/gemma-4-26b-a4b")
+
+# DSPy uses LiteLLM-style model strings. "openai/" prefix routes through the
+# OpenAI-compatible client, which LM Studio speaks.
+TASK_MODEL_STRING = f"openai/{LOCAL_MODEL}"
+
+TASK_LM_KWARGS = dict(
+    api_base=LM_STUDIO_BASE,
+    api_key=os.getenv("LM_STUDIO_KEY", "lm-studio"),  # any non-empty string
+    temperature=0.6,
+    max_tokens=4096,  # ChainOfThought reasoning + all output fields easily exceeds 2k
+    cache=True,
+)
+
+# Which backend to use: "gemini" if the API key is present, else "lm_studio".
+# Override with TASK_LM_BACKEND=lm_studio to force local even when the key is set.
+TASK_LM_BACKEND: str = os.getenv("TASK_LM_BACKEND", "gemini" if GEMINI_API_KEY else "lm_studio")
+
+
+# ──────────────────────────── Enrichment LM (OpenAI gpt-4o-mini, offline batch) ─────────
+# gpt-4o-mini is reliable at structured JSON output and cheap enough that the
+# full 701-verse corpus costs under $1 (one-time).
+#
+# Cost estimate (full 701-verse corpus):
+#   ~1800 input tokens/verse × 701 × $0.15/M ≈ $0.19 input
+#   ~900  output tokens/verse × 701 × $0.60/M ≈ $0.38 output
+#   Total ≈ $0.57 — effectively free at this scale.
+#
+# Key is read from .env (OPENAI_API_KEY). Override ENRICH_MODEL env var to
+# swap in a different OpenAI model (e.g. "openai/gpt-4o" for harder cases).
+ENRICH_MODEL = os.getenv("ENRICH_MODEL", "openai/gpt-4o-mini")
+
+ENRICH_LM_KWARGS = dict(
+    api_key=os.getenv("OPENAI_API_KEY", ""),
+    temperature=0.3,   # lower than task LM — we want consistent structured output
+    max_tokens=3000,   # enough headroom for all six fields + CoT reasoning
+    cache=True,        # DSPy disk cache deduplicates identical calls on re-runs
+    response_format={"type": "text"},  # DSPy 3.x sends json_object by default;
+                                       # OpenAI now requires json_schema or text
+)
+
+
+# ──────────────────────────── Proxy Task LM (gpt-4o-mini, GEPA optimization only) ────────
+# When running GEPA with --proxy-task-lm, this model replaces Gemma 4 as the task LM
+# during optimization. Prompts are model-agnostic text; they transfer back to Gemma 4
+# at inference time. gpt-4o-mini runs ~20x faster than Gemma 4 thinking mode, bringing
+# --auto light from ~260 hours to ~2-3 hours.
+PROXY_TASK_MODEL = os.getenv("PROXY_TASK_MODEL", "openai/gpt-4o-mini")
+
+PROXY_TASK_LM_KWARGS = dict(
+    api_key=os.getenv("OPENAI_API_KEY", ""),
+    temperature=0.6,
+    max_tokens=4096,
+    cache=True,
+    response_format={"type": "text"},
+)
+
+# ──────────────────────────── Reflection LM (gpt-4o, GEPA) ──────────────────────────────
+# GEPA's reflection step reads metric feedback and proposes rewritten prompts.
+# This scales strongly with model quality. gpt-4o is the right balance here:
+# it reasons well enough to write meaningful prompt mutations from nuanced
+# Advaita feedback, and is affordable on a small OpenAI credit balance.
+#
+# Cost estimate per GEPA run (reflection calls only):
+#   --auto light:  ~50 calls × 6k tokens ≈ $1.50
+#   --auto medium: ~250 calls × 6k tokens ≈ $7.50
+#
+# gpt-4o-mini is too shallow for this task — it produces generic rewrites
+# that ignore the tradition-specific feedback the metric provides.
+# Same OPENAI_API_KEY as the enrichment LM.
+REFLECTION_MODEL = os.getenv("REFLECTION_MODEL", "openai/gpt-4o")
+
+REFLECTION_LM_KWARGS = dict(
+    api_key=os.getenv("OPENAI_API_KEY", ""),
+    temperature=1.0,   # GEPA wants diversity across reflection proposals
+    max_tokens=6000,   # headroom for detailed critique + full rewritten prompt text
+    response_format={"type": "text"},  # same fix as enrichment LM — avoid json_object
+    cache=False,       # reflection calls are intentionally diverse; caching defeats that
+)
+
+
+# ──────────────────────────── Configure helpers ───────────────────────────────────────
+def configure_dspy() -> tuple[dspy.LM, dspy.LM]:
+    """Configure DSPy for inference and return (task_lm, reflection_lm).
+
+    Prefers Gemini API when GEMINI_API_KEY is set (same Gemma 4 26B weights,
+    hosted by Google, free tier). Falls back to LM Studio otherwise.
+    Override with TASK_LM_BACKEND=lm_studio env var to force local.
+
+    ChatAdapter fallback to JSONAdapter is disabled in both paths because:
+    - LM Studio rejects json_object.
+    - Gemma outputs `[[ ## field ]]` (no closing ##); the field_header_pattern
+      patch at module load time makes ChatAdapter parse these correctly.
+    """
+    if TASK_LM_BACKEND == "gemini":
+        task_lm = dspy.LM(model=GEMINI_TASK_MODEL, **GEMINI_TASK_LM_KWARGS)
+        print(f"Task LM backend: Gemini API ({GEMINI_TASK_MODEL})")
+    else:
+        task_lm = dspy.LM(model=TASK_MODEL_STRING, **TASK_LM_KWARGS)
+        print(f"Task LM backend: LM Studio ({TASK_MODEL_STRING} @ {LM_STUDIO_BASE})")
+
+    reflection_lm = dspy.LM(model=REFLECTION_MODEL, **REFLECTION_LM_KWARGS)
+    # use_json_adapter_fallback=False: LM Studio rejects json_object, so we must never fall back
+    dspy.configure(lm=task_lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False))
+    return task_lm, reflection_lm
+
+
+def configure_enrich_lm() -> dspy.LM:
+    """Configure DSPy globally with the Claude Sonnet enrichment LM and return it.
+
+    Call this instead of configure_dspy() when running enrich_corpus.py.
+    Raises if ANTHROPIC_API_KEY is not set.
+    """
+    key = os.getenv("OPENAI_API_KEY", "")
+    if not key:
+        raise SystemExit(
+            "OPENAI_API_KEY is not set. Add it to your .env file:\n"
+            "  OPENAI_API_KEY=sk-proj-..."
+        )
+    lm = dspy.LM(model=ENRICH_MODEL, **ENRICH_LM_KWARGS)
+    dspy.configure(lm=lm)
+    return lm
+
+
+# ──────────────────────────── Embeddings ─────────────────────────────────────────────
+# Local sentence-transformer for retrieval. BGE-small is a sweet spot for
+# semantic philosophy text on a Mac without burning RAM.
+EMBED_MODEL = os.getenv("EMBED_MODEL", "BAAI/bge-small-en-v1.5")
+EMBED_DEVICE = os.getenv("EMBED_DEVICE", "mps")  # "mps" on Apple Silicon, "cpu" otherwise
+
+TOP_K_RETRIEVE = 8       # passages to fetch per query
+N_RETRIEVAL_QUERIES = 3  # the planner generates this many per user question

corpus.py

@@ -0,0 +1,224 @@
+"""
+corpus.py — the data model and on-disk storage for the verse corpus.
+
+A note on dataclasses vs. plain dicts
+-------------------------------------
+We could have used dicts everywhere and saved keystrokes. We don't, because
+the Verse type is the contract between five different modules — parsers,
+enrichment, indexing, retrieval, and the metric — and a typed contract
+catches mistakes that "I thought 'sources_cited' was a list" wouldn't.
+
+The pipeline lifecycle of a verse
+---------------------------------
+    parsers.*        →  Verse           (no LLM-derived fields)
+    enrichment.py    →  EnrichedVerse   (with LLM-derived fields)
+    knowledge_base   →  reads EnrichedVerse, writes 3 embeddings per verse
+    advisor.py       →  receives EnrichedVerse via retriever hits
+    metrics.py       →  uses verse_id for exact citation grounding
+
+Storage choice
+--------------
+JSONL on disk. Each line is a verse. Why not Parquet, sqlite, etc.?
+- Easy to grep
+- Easy to diff in PRs
+- Easy for a human to spot-check enrichment quality (the whole point)
+- We never need to scan more than a few thousand lines, so format doesn't matter
+"""
+
+from __future__ import annotations
+import json
+from dataclasses import dataclass, field, asdict, fields
+from pathlib import Path
+from typing import Iterable, Iterator
+
+
+# ──────────────────────────── Verse: the raw record ────────────────────────────
+@dataclass
+class Verse:
+    """A natural unit of scripture: one verse, one mantra, one sūtra.
+
+    The required fields are minimal — every parser must produce at least these.
+    Optional fields (sanskrit, transliteration, bhashya, ...) are filled when
+    the source provides them.
+
+    `verse_id` is the global unique key. Convention:
+        '<work_slug>_<section_slug>_<verse_number>'
+    e.g. 'bhagavad_gita_02_47', 'mundaka_upanishad_2_1_3'.
+
+    `verse_ref` is the human-readable citation form:
+        e.g. 'BG 2.47', 'Muṇḍaka Up. 2.1.3', 'Vivekacūḍāmaṇi 11'.
+    The advisor's response uses this exact string in citations.
+    """
+    # Identity — required for every record
+    verse_id: str
+    work: str
+    work_display: str
+    verse_ref: str
+    tier: str                         # primary | shankara | supporting
+
+    # Section/chapter info — required when the work has chapters
+    section: str = ""                 # 'chapter_02'
+    section_display: str = ""         # 'Chapter 2: Sāṅkhya Yoga'
+
+    # Content — at least one of {translation, bhashya} must be non-empty
+    translation: str = ""             # English translation of the verse itself
+    translator: str = ""              # who translated it (for attribution)
+
+    sanskrit: str = ""                # original Devanāgarī
+    transliteration: str = ""         # IAST roman transliteration
+    word_meanings: str = ""           # word-by-word gloss when present
+
+    bhashya: str = ""                 # Śaṅkara's commentary on this verse, if any
+    bhashya_translator: str = ""      # who translated the bhāṣya
+
+    # Provenance for accountability and license display
+    source_key: str = ""              # the registry key this came from
+    license: str = ""                 # license tag from registry
+
+    def has_content(self) -> bool:
+        """Used by parsers/loaders to drop empty records before they pollute
+        the index. A 'verse' with only a verse_id and no actual text is junk."""
+        return bool(self.translation.strip() or self.bhashya.strip())
+
+
+# ──────────────────────────── EnrichedVerse: with LLM extractions ────────────────
+@dataclass
+class EnrichedVerse(Verse):
+    """A Verse + the structured fields produced by the offline LLM pass.
+
+    Every list defaults to empty so a verse that fails enrichment can still
+    be stored (without enrichment, indexed only on its literal text/bhāṣya).
+    """
+    # The plain-English statement of what the verse teaches. Ideally 1–2
+    # sentences. This is what the synthesizer reads downstream.
+    paraphrase: str = ""
+
+    # Vedānta concepts engaged by the verse. Tradition-native vocabulary.
+    # Examples: 'karma_yoga', 'vairagya', 'sakshi', 'two_truths', 'adhyasa'.
+    themes: list[str] = field(default_factory=list)
+
+    # Mundane life situations where this verse would help. User-language.
+    # Examples: 'facing failure after sustained effort', 'watching a parent decline'.
+    life_situations: list[str] = field(default_factory=list)
+
+    # Emotions addressed, from a small consistent vocabulary.
+    # See enrichment.py EMOTION_VOCAB for the closed set.
+    emotions_addressed: list[str] = field(default_factory=list)
+
+    # What does this verse ask the seeker to do or shift?
+    practical_teaching: str = ""
+
+    # Hypothetical questions a real person might bring to this verse.
+    # These are gold for retrieval; they bridge the language gap.
+    hypothetical_questions: list[str] = field(default_factory=list)
+
+    # Quality / debugging
+    enrichment_model: str = ""        # which LM produced these fields
+    enrichment_version: int = 1       # bump when the prompt changes substantively
+
+    # ---- Derived "views" used at indexing time ----
+    def literal_view(self) -> str:
+        """The literal English translation, lightly enriched with the Sanskrit
+        if available. Best for queries that share lexical features with the text."""
+        parts = []
+        if self.translation:
+            parts.append(self.translation.strip())
+        if self.transliteration:
+            parts.append(f"({self.transliteration.strip()})")
+        return "\n".join(parts)
+
+    def bhashya_view(self) -> str:
+        """Śaṅkara's commentary on this verse. Best for queries about the
+        Vedāntic explanation rather than the verse text itself."""
+        return self.bhashya.strip()
+
+    def advisor_view(self) -> str:
+        """The composed view that bridges the language gap.
+
+        This is what makes the user-question-→-verse mapping work. A user who
+        types 'I feel hollow even though I got everything I wanted' will not
+        find anything in the Sanskrit. They will find a near-neighbor in this
+        view if the enrichment did its job.
+        """
+        bits = []
+        if self.paraphrase:
+            bits.append(f"Teaching: {self.paraphrase}")
+        if self.life_situations:
+            bits.append(
+                "Speaks to: " + "; ".join(self.life_situations)
+            )
+        if self.emotions_addressed:
+            bits.append(
+                "Addresses: " + ", ".join(self.emotions_addressed)
+            )
+        if self.themes:
+            bits.append(
+                "Themes: " + ", ".join(self.themes)
+            )
+        if self.hypothetical_questions:
+            bits.append(
+                "Questions this answers:\n  - "
+                + "\n  - ".join(self.hypothetical_questions)
+            )
+        if self.practical_teaching:
+            bits.append(f"Practical shift: {self.practical_teaching}")
+        return "\n".join(bits)
+
+    def is_enriched(self) -> bool:
+        """Did enrichment populate at least the minimum-viable fields?"""
+        return bool(self.paraphrase) and bool(self.life_situations) and bool(self.hypothetical_questions)
+
+
+# ──────────────────────────── On-disk JSONL ────────────────────────────
+def write_jsonl(records: Iterable[Verse], path: Path) -> int:
+    """Write a stream of records as JSONL. Returns count written."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    n = 0
+    with path.open("w", encoding="utf-8") as f:
+        for r in records:
+            f.write(json.dumps(asdict(r), ensure_ascii=False) + "\n")
+            n += 1
+    return n
+
+
+def read_jsonl_verses(path: Path) -> Iterator[Verse]:
+    """Read a JSONL file as Verse records. Skips lines we can't parse."""
+    if not path.exists():
+        return
+    with path.open(encoding="utf-8") as f:
+        for line_no, line in enumerate(f, start=1):
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                d = json.loads(line)
+                yield _verse_from_dict(d, Verse)
+            except Exception as e:
+                print(f"[corpus] skipping malformed line {line_no} in {path}: {e}")
+
+
+def read_jsonl_enriched(path: Path) -> Iterator[EnrichedVerse]:
+    """Read a JSONL file as EnrichedVerse records."""
+    if not path.exists():
+        return
+    with path.open(encoding="utf-8") as f:
+        for line_no, line in enumerate(f, start=1):
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                d = json.loads(line)
+                yield _verse_from_dict(d, EnrichedVerse)
+            except Exception as e:
+                print(f"[corpus] skipping malformed line {line_no} in {path}: {e}")
+
+
+def _verse_from_dict(d: dict, cls):
+    """Construct a Verse/EnrichedVerse, ignoring keys the dataclass doesn't know.
+
+    This forward-compatibility matters: if a future version adds a field, old
+    JSONL files should still load. And if enrichment adds extra debug fields,
+    we don't want the dataclass to choke on them.
+    """
+    valid = {f.name for f in fields(cls)}
+    return cls(**{k: v for k, v in d.items() if k in valid})

dataset_generator.py

@@ -0,0 +1,332 @@
+"""
+dataset_generator.py — produce ~500 unique, life-grounded questions.
+
+The dataset is the GEPA training/validation pool. We want:
+  - Coverage across life domains (career, grief, identity, dharma, practice, ...)
+  - Variety in voice (anguished / intellectual / sarcastic / exhausted / hopeful)
+  - Variety in form (direct question / vent / philosophical doubt / dilemma)
+  - Variety in age & life-stage cues
+  - Some cleanly Advaita-relevant, some that *force* the advisor to find the
+    Advaita angle in something mundane (this is where over-fitting to "spiritual"
+    questions usually shows up)
+
+Strategy: structured combinatorics × LM rewriting × similarity dedupe.
+
+We construct (domain, scenario, voice, form) tuples, send them to the local LM
+to write each as a real human message, then dedupe by embedding similarity.
+"""
+
+from __future__ import annotations
+import argparse
+import json
+import random
+import re
+from dataclasses import dataclass, asdict
+from pathlib import Path
+
+import numpy as np
+from sentence_transformers import SentenceTransformer
+from tqdm import tqdm
+import dspy
+
+import config
+
+
+# ──────────────────────────── Taxonomy ────────────────────────────
+DOMAINS: dict[str, list[str]] = {
+    "career_and_purpose": [
+        "got laid off after years of dedication",
+        "achieved the big career goal and feels empty",
+        "stuck in a job that pays well but feels meaningless",
+        "wants to leave stable career to pursue art / spiritual path",
+        "watching peers succeed while their own work plateaus",
+        "facing retirement and loss of identity tied to work",
+        "imposter syndrome after a major promotion",
+        "publicly failed in front of colleagues",
+    ],
+    "romantic_relationships": [
+        "going through a painful breakup after long relationship",
+        "marriage has gone cold and considering divorce",
+        "in love with someone who doesn't love them back",
+        "obsessive jealousy about a partner's past",
+        "tempted to have an affair",
+        "partner died and grief is overwhelming",
+        "afraid of commitment despite loving partner",
+        "single in their 40s and despairing about it",
+    ],
+    "family": [
+        "parent is dying and they have unresolved conflict",
+        "estranged from a sibling for years",
+        "parents pressuring them about marriage / career",
+        "child making destructive life choices",
+        "caring for an aging parent and exhausted",
+        "had a falling out with adult child",
+        "mother-in-law conflict ruining marriage",
+        "feels they failed as a parent",
+    ],
+    "friendship_and_social": [
+        "best friend betrayed their trust",
+        "feels invisible and lonely in their 30s",
+        "friend group has drifted apart with age",
+        "social anxiety preventing them from connecting",
+        "outgrown their old friends spiritually",
+        "discovered close friend was talking behind their back",
+    ],
+    "mortality_and_loss": [
+        "received a serious medical diagnosis",
+        "watching a loved one die slowly",
+        "afraid of death after a near-miss",
+        "grieving a sudden, unexpected loss",
+        "watching parents age and decline",
+        "lost a child",
+        "lost a pet who was their closest companion",
+        "approaching old age with regret about unlived life",
+    ],
+    "identity_and_ego": [
+        "tying self-worth entirely to external validation",
+        "endlessly comparing themselves to others on social media",
+        "going through midlife crisis questioning everything",
+        "famous and feels everyone wants something from them",
+        "lost sense of who they are after big life change",
+        "racial / cultural identity feels splintered between worlds",
+        "transitioning gender and family rejecting them",
+    ],
+    "material_life": [
+        "drowning in debt and shame about it",
+        "wealthy and feels guilty / disconnected because of it",
+        "consumed by FOMO scrolling through richer friends' lives",
+        "lost their home / financial security",
+        "struggling to give up consumerist habits despite knowing better",
+        "tempted by a get-rich-quick scheme",
+    ],
+    "existential": [
+        "feels life has no meaning at all",
+        "deeply depressed and going through the motions",
+        "constant existential dread about the world's state",
+        "doubting whether God / Brahman exists",
+        "sees through everything and now nothing feels real",
+        "feels they were 'born wrong' for this world",
+    ],
+    "spiritual_practice": [
+        "meditation has gone dry after years of practice",
+        "got addicted to spiritual highs and now they've stopped",
+        "spiritual ego — feels superior to non-practitioners",
+        "had a powerful experience and can't get back to it",
+        "doubts whether their guru / lineage is right for them",
+        "intellectually understands non-duality but doesn't feel it",
+        "afraid that liberation means losing love for family",
+        "can't reconcile traditional teachings with modern life",
+    ],
+    "ethics_and_dharma": [
+        "told a serious lie and considering whether to confess",
+        "harmed someone in the past and can't forgive themselves",
+        "facing a moral dilemma at work involving dishonesty",
+        "tempted to retaliate against someone who wronged them",
+        "torn between duty to family and personal calling",
+        "did something they're deeply ashamed of",
+    ],
+    "health_and_body": [
+        "chronic illness reshaping their entire life",
+        "struggling with addiction and relapse",
+        "eating disorder they can't seem to escape",
+        "chronic pain making spiritual practice feel impossible",
+        "hates their aging body",
+        "cancer diagnosis reframing everything",
+    ],
+    "modernity_specific": [
+        "doomscrolling and feeling worse every day",
+        "AI / automation making them feel obsolete",
+        "climate dread paralyzing their life decisions",
+        "political division has destroyed family relationships",
+        "addicted to phone / can't focus / can't read books anymore",
+        "online persona feels disconnected from real self",
+    ],
+}
+
+VOICES = [
+    "anguished",
+    "exhausted",
+    "intellectual and analytical",
+    "darkly sarcastic",
+    "quietly hopeful",
+    "numb and dissociated",
+    "frustrated and angry",
+    "softly resigned",
+]
+
+FORMS = [
+    "direct question",
+    "venting paragraph",
+    "philosophical doubt",
+    "practical dilemma asking what to do",
+    "stream-of-consciousness",
+]
+
+AGE_CUES = [
+    "early 20s",
+    "late 20s",
+    "early 30s",
+    "late 30s",
+    "40s",
+    "50s",
+    "60s",
+    "70s",
+    "(no age cue)",
+]
+
+
+@dataclass
+class QuestionRecord:
+    id: str
+    question: str
+    domain: str
+    scenario: str
+    voice: str
+    form: str
+    age_cue: str
+
+
+# ──────────────────────────── LM-driven phrasing ────────────────────────────
+class WriteUserMessage(dspy.Signature):
+    """Write a single, realistic message that a person might send to a spiritual
+    advisor. The message must reflect the given scenario, voice, form, and age
+    cue. Do NOT include scripture references, do NOT name Vedānta concepts —
+    write as a real person speaking from their actual life. Avoid generic phrases
+    like 'help me find peace' or 'I want to grow spiritually'. Be specific, lived,
+    grounded in detail. 2-6 sentences."""
+
+    scenario: str = dspy.InputField()
+    voice: str = dspy.InputField()
+    form: str = dspy.InputField()
+    age_cue: str = dspy.InputField()
+
+    message: str = dspy.OutputField(desc="The user's message, in first person.")
+
+
+def _slug(s: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "_", s.lower()).strip("_")[:60]
+
+
+def generate_questions(target_n: int = 500, seed: int = 7, use_local: bool = False) -> list[QuestionRecord]:
+    """Generate ~target_n unique questions via combinatorics + LM rewriting."""
+    rng = random.Random(seed)
+    if use_local:
+        config.configure_dspy()
+    else:
+        config.configure_enrich_lm()  # gpt-4o-mini: faster and more stylistically diverse
+    writer = dspy.Predict(WriteUserMessage)
+
+    # Build the (domain, scenario, voice, form, age) plan first
+    combos: list[tuple[str, str, str, str, str]] = []
+    for domain, scenarios in DOMAINS.items():
+        for scenario in scenarios:
+            # 5 variants per scenario varying voice/form/age
+            voices = rng.sample(VOICES, k=5)
+            forms = [rng.choice(FORMS) for _ in range(5)]
+            ages = rng.sample(AGE_CUES, k=5)
+            for v, f, a in zip(voices, forms, ages):
+                combos.append((domain, scenario, v, f, a))
+
+    rng.shuffle(combos)
+
+    # Cap to a generous over-target; we'll dedupe down to target_n
+    over_target = int(target_n * 1.25)
+    combos = combos[:over_target]
+
+    records: list[QuestionRecord] = []
+    for i, (domain, scenario, voice, form, age) in enumerate(tqdm(combos, desc="Generating")):
+        try:
+            out = writer(scenario=scenario, voice=voice, form=form, age_cue=age)
+            msg = (out.message or "").strip()
+            if len(msg) < 30:
+                continue
+            records.append(QuestionRecord(
+                id=f"q_{i:04d}_{_slug(domain)}",
+                question=msg,
+                domain=domain,
+                scenario=scenario,
+                voice=voice,
+                form=form,
+                age_cue=age,
+            ))
+        except Exception as e:
+            # Local LMs occasionally hiccup. Log and continue.
+            print(f"[warn] generation failure on combo {i}: {e}")
+            continue
+
+    return _dedupe_by_similarity(records, target_n=target_n)
+
+
+def _dedupe_by_similarity(records: list[QuestionRecord], target_n: int, threshold: float = 0.92) -> list[QuestionRecord]:
+    """Embed and remove near-duplicates greedily."""
+    if not records:
+        return records
+    print(f"Deduping {len(records)} candidates ...")
+    embedder = SentenceTransformer(config.EMBED_MODEL, device=config.EMBED_DEVICE)
+    embs = embedder.encode(
+        [r.question for r in records],
+        normalize_embeddings=True,
+        show_progress_bar=True,
+        batch_size=32,
+    )
+    keep_idx: list[int] = []
+    kept_embs = []
+    for i, e in enumerate(embs):
+        if not kept_embs:
+            keep_idx.append(i)
+            kept_embs.append(e)
+            continue
+        sims = np.dot(np.stack(kept_embs), e)
+        if float(sims.max()) < threshold:
+            keep_idx.append(i)
+            kept_embs.append(e)
+        if len(keep_idx) >= target_n:
+            break
+    print(f"Kept {len(keep_idx)} after dedupe (target {target_n}).")
+    return [records[i] for i in keep_idx]
+
+
+def save_jsonl(records: list[QuestionRecord], path: Path):
+    with path.open("w", encoding="utf-8") as f:
+        for r in records:
+            f.write(json.dumps(asdict(r), ensure_ascii=False) + "\n")
+    print(f"Wrote {len(records)} questions to {path}")
+
+
+def load_jsonl(path: Path = config.DATASET_PATH) -> list[dict]:
+    with path.open(encoding="utf-8") as f:
+        return [json.loads(line) for line in f if line.strip()]
+
+
+def to_dspy_examples(records: list[dict]) -> list[dspy.Example]:
+    """The dataset has no gold labels — that's fine. GEPA's metric uses LLM
+    judgment + retrieval grounding rather than reference answers.
+    We carry the metadata as inputs-of-record so the metric can use them."""
+    out = []
+    for r in records:
+        ex = dspy.Example(
+            user_question=r["question"],
+            history=dspy.History(messages=[]),
+            domain=r["domain"],
+            scenario=r["scenario"],
+        ).with_inputs("user_question", "history")
+        out.append(ex)
+    return out
+
+
+# ──────────────────────────── CLI ────────────────────────────
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--n", type=int, default=500)
+    ap.add_argument("--seed", type=int, default=7)
+    ap.add_argument("--out", type=str, default=str(config.DATASET_PATH))
+    ap.add_argument("--lm", choices=["openai", "local"], default="openai",
+                    help="openai = gpt-4o-mini (default, faster); local = LM Studio task LM")
+    args = ap.parse_args()
+
+    records = generate_questions(target_n=args.n, seed=args.seed, use_local=(args.lm == "local"))
+    save_jsonl(records, Path(args.out))
+
+
+if __name__ == "__main__":
+    main()

download_sources.py

@@ -0,0 +1,195 @@
+"""
+download_sources.py — fetch every enabled source from the registry.
+
+What this does
+--------------
+Reads sources_registry.SOURCES, walks each enabled entry, and downloads its
+files into data/raw/<source_key>/. The downloader is deliberately dumb: it
+just gets the bytes onto disk. Parsing happens in a separate step (parsers/)
+so a download failure on one source doesn't block ingest of the others, and
+so re-parsing during prompt iteration doesn't re-hit the network.
+
+Why HTTPS over `requests` rather than git for everything
+--------------------------------------------------------
+Most of our sources are individual JSON or HTML files. Cloning a whole repo
+to get two files wastes bandwidth and brittle-ifies the script. For sources
+that *are* whole repos (rare in our registry), prefix the URL with `git+`.
+
+Idempotency
+-----------
+If a file is already present and not corrupt, we skip it. Pass --force to
+re-download. This makes it safe to run repeatedly while debugging parsers.
+
+Politeness
+----------
+We send a real User-Agent and rate-limit to one request per second per host.
+Internet Archive and similar mirrors are gracious to projects that play nice;
+they can also throttle aggressively when they aren't.
+"""
+
+from __future__ import annotations
+import argparse
+import shutil
+import subprocess
+import sys
+import time
+from collections import defaultdict
+from pathlib import Path
+from urllib.parse import urlparse
+
+import requests
+from tqdm import tqdm
+
+import config
+from sources_registry import SOURCES, Source
+
+
+RAW_DIR = config.DATA_DIR / "raw"
+USER_AGENT = (
+    "GitaAdvisor/0.2 (Advaita-Vedanta research project; "
+    "contact: <add your email here>)"
+)
+
+# Per-host minimum interval in seconds
+MIN_INTERVAL = 1.0
+
+
+def _filename_for_url(url: str) -> str:
+    """Derive a sensible local filename from a URL."""
+    parsed = urlparse(url)
+    name = Path(parsed.path).name or "index.html"
+    # archive.org sometimes serves djvu.txt with no extension on the URL;
+    # keep what's there.
+    return name
+
+
+def _is_git_url(url: str) -> bool:
+    return url.startswith("git+")
+
+
+_last_request_time: dict = defaultdict(float)
+
+
+def _polite_get(url: str) -> requests.Response:
+    """GET with rate limiting per host."""
+    host = urlparse(url).netloc
+    elapsed = time.time() - _last_request_time[host]
+    if elapsed < MIN_INTERVAL:
+        time.sleep(MIN_INTERVAL - elapsed)
+    _last_request_time[host] = time.time()
+    return requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=60, stream=True)
+
+
+def _download_file(url: str, dest: Path, force: bool = False) -> bool:
+    """Download a single URL to dest. Returns True if a download happened
+    (vs being skipped because already present)."""
+    if dest.exists() and dest.stat().st_size > 0 and not force:
+        return False
+
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    tmp = dest.with_suffix(dest.suffix + ".tmp")
+
+    with _polite_get(url) as r:
+        r.raise_for_status()
+        total = int(r.headers.get("content-length", 0)) or None
+        with tmp.open("wb") as out, tqdm(
+            total=total, unit="B", unit_scale=True, leave=False, desc=dest.name
+        ) as bar:
+            for chunk in r.iter_content(chunk_size=8192):
+                if not chunk:
+                    continue
+                out.write(chunk)
+                bar.update(len(chunk))
+
+    tmp.replace(dest)
+    return True
+
+
+def _clone_git(url: str, dest_dir: Path, force: bool = False) -> bool:
+    """Clone a git repo (URL prefixed with 'git+') into dest_dir. Returns
+    True if a clone happened."""
+    real_url = url[len("git+"):]
+    if dest_dir.exists() and any(dest_dir.iterdir()) and not force:
+        return False
+    if dest_dir.exists():
+        shutil.rmtree(dest_dir)
+    dest_dir.parent.mkdir(parents=True, exist_ok=True)
+    subprocess.run(
+        ["git", "clone", "--depth=1", real_url, str(dest_dir)],
+        check=True,
+    )
+    return True
+
+
+def download_source(src: Source, force: bool = False) -> dict:
+    """Download all URLs for one source. Returns a small report dict."""
+    target = RAW_DIR / src.key
+    report = {"key": src.key, "ok": 0, "skipped": 0, "failed": []}
+
+    if not src.urls:
+        report["failed"].append("no URLs in registry entry")
+        return report
+
+    for url in src.urls:
+        if not url:
+            continue
+        try:
+            if _is_git_url(url):
+                changed = _clone_git(url, target, force=force)
+            else:
+                fname = _filename_for_url(url)
+                changed = _download_file(url, target / fname, force=force)
+            if changed:
+                report["ok"] += 1
+            else:
+                report["skipped"] += 1
+        except Exception as e:
+            report["failed"].append(f"{url}: {e}")
+    return report
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Download all enabled sources from the registry.")
+    ap.add_argument("--force", action="store_true",
+                    help="Re-download even if files exist.")
+    ap.add_argument("--only", nargs="*", default=None,
+                    help="Only download these source keys.")
+    args = ap.parse_args()
+
+    enabled = [s for s in SOURCES if s.enabled]
+    if args.only:
+        enabled = [s for s in enabled if s.key in set(args.only)]
+    if not enabled:
+        print("No enabled sources match. Edit sources_registry.py to enable some.")
+        sys.exit(1)
+
+    print(f"Downloading {len(enabled)} sources to {RAW_DIR}")
+    print(f"User-Agent: {USER_AGENT}")
+    print()
+
+    any_failed = False
+    for src in enabled:
+        print(f"━━━ {src.key} — {src.name}")
+        print(f"    license={src.license}  tier={src.tier}  parser={src.parser}")
+        if src.translator:
+            year = f", {src.year}" if src.year else ""
+            print(f"    translator: {src.translator}{year}")
+
+        report = download_source(src, force=args.force)
+        if report["failed"]:
+            any_failed = True
+            for f in report["failed"]:
+                print(f"    [FAIL] {f}")
+        print(f"    downloaded={report['ok']}  cached={report['skipped']}")
+        print()
+
+    if any_failed:
+        print("Some sources failed. Re-run with the network available, or "
+              "edit the URL in sources_registry.py if a mirror has moved.")
+        sys.exit(2)
+    print("All enabled sources are now on disk under data/raw/.")
+    print("Next: python ingest_corpus.py")
+
+
+if __name__ == "__main__":
+    main()

enrich_corpus.py

@@ -0,0 +1,174 @@
+"""
+enrich_corpus.py — run the local LLM over every verse, once, with caching.
+
+The cost calculus
+-----------------
+For ~3,000 verses at ~30s per call on a 26B-class local model, a full pass
+takes a long evening — call it 25 hours. That's tolerable as a one-time cost,
+intolerable as a recurring one. So caching is non-negotiable. We cache by
+verse_id and the enrichment_version stamp; if you change the prompt
+substantively, bump the version in enrichment.py and the next run re-enriches.
+
+What we write
+-------------
+data/corpus_enriched.jsonl — one EnrichedVerse per line, in the same order
+as data/corpus.jsonl. Failed enrichments are still written (with empty
+enrichment fields and an error stamp in enrichment_model) so the index can
+still cover them on their literal text.
+
+Concurrency
+-----------
+LM Studio's OpenAI-compatible server processes requests serially by default.
+We don't try to parallelize at the client; if you've configured your server
+for parallel decode, set --concurrency > 1 and DSPy will hold multiple
+in-flight calls. For modest hardware, 1 is correct.
+
+Resumability
+------------
+If the run dies halfway, just re-run. The cache at data/enrichment_cache.jsonl
+remembers per-verse what we already did, so we pick up exactly where we left
+off. No flag is needed for resume; it's the default behavior.
+"""
+
+from __future__ import annotations
+import argparse
+import json
+import os
+from dataclasses import asdict
+from pathlib import Path
+from typing import Iterable
+
+from tqdm import tqdm
+import dspy
+
+import config
+from corpus import Verse, EnrichedVerse, read_jsonl_verses, write_jsonl
+from enrichment import Enricher
+
+
+CACHE_PATH = config.DATA_DIR / "enrichment_cache.jsonl"
+ENRICHED_PATH = config.DATA_DIR / "corpus_enriched.jsonl"
+
+
+# ──────────────────────────── Cache I/O ────────────────────────────
+def _load_cache(path: Path) -> dict[str, EnrichedVerse]:
+    """Load cache as {verse_id: EnrichedVerse}. Tolerates partial writes."""
+    if not path.exists():
+        return {}
+    out: dict[str, EnrichedVerse] = {}
+    with path.open(encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                d = json.loads(line)
+                ev = EnrichedVerse(**{k: v for k, v in d.items() if k in EnrichedVerse.__dataclass_fields__})
+                out[ev.verse_id] = ev
+            except Exception:
+                continue
+    return out
+
+
+def _append_cache(path: Path, ev: EnrichedVerse) -> None:
+    """Append a single record. We use append-mode rather than rewriting so
+    a kill -9 mid-run loses at most one line."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("a", encoding="utf-8") as f:
+        f.write(json.dumps(asdict(ev), ensure_ascii=False) + "\n")
+
+
+# ──────────────────────────── Main loop ────────────────────────────
+def enrich_all(
+    in_path: Path,
+    out_path: Path,
+    cache_path: Path,
+    limit: int | None = None,
+    re_enrich: bool = False,
+    only_failed: bool = False,
+    use_claude: bool = True,
+) -> None:
+    if use_claude:
+        lm = config.configure_enrich_lm()
+        print(f"[enrich] LM: {lm.model} (Claude API)")
+    else:
+        config.configure_dspy()
+        print(f"[enrich] LM: {config.LOCAL_MODEL} (local LM Studio)")
+    enricher = Enricher()
+
+    cache = _load_cache(cache_path) if not re_enrich else {}
+    print(f"[enrich] cache contains {len(cache)} previously-enriched verses")
+
+    verses = list(read_jsonl_verses(in_path))
+    if limit:
+        verses = verses[:limit]
+    print(f"[enrich] enriching {len(verses)} verses from {in_path}")
+
+    enriched: list[EnrichedVerse] = []
+    pending = []
+    for v in verses:
+        cached = cache.get(v.verse_id)
+        if cached and not re_enrich:
+            if only_failed and cached.enrichment_model.startswith("FAILED"):
+                pending.append(v)
+            else:
+                enriched.append(cached)
+                continue
+        else:
+            pending.append(v)
+
+    print(f"[enrich] {len(enriched)} from cache, {len(pending)} to call LM for")
+
+    n_failed = 0
+    for v in tqdm(pending, desc="enriching"):
+        ev = enricher(verse=v)
+        _append_cache(cache_path, ev)
+        enriched.append(ev)
+        if not ev.is_enriched():
+            n_failed += 1
+
+    # Restore original verse order from in_path
+    by_id = {ev.verse_id: ev for ev in enriched}
+    ordered = [by_id[v.verse_id] for v in verses if v.verse_id in by_id]
+
+    n_written = write_jsonl(ordered, out_path)
+    print(f"[enrich] wrote {n_written} enriched verses to {out_path}")
+    if n_failed:
+        print(f"[enrich] WARNING: {n_failed} verses failed enrichment "
+              f"(empty fields, indexed only on literal text). "
+              f"Re-run with --only-failed to retry just those.")
+
+
+# ──────────────────────────── CLI ────────────────────────────
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--in", dest="in_path",
+                    default=str(config.DATA_DIR / "corpus.jsonl"))
+    ap.add_argument("--out", default=str(ENRICHED_PATH))
+    ap.add_argument("--cache", default=str(CACHE_PATH))
+    ap.add_argument("--limit", type=int, default=None,
+                    help="Enrich only the first N verses (smoke-test).")
+    ap.add_argument("--re-enrich", action="store_true",
+                    help="Ignore cache and re-enrich everything. Use this "
+                         "when you change the enrichment prompt.")
+    ap.add_argument("--only-failed", action="store_true",
+                    help="Re-run only the verses whose previous enrichment "
+                         "failed (FAILED stamp in enrichment_model).")
+    ap.add_argument("--lm", choices=["claude", "local"], default="claude",
+                    help="Which LM to use: 'claude' (default, Sonnet 4.6 via API) "
+                         "or 'local' (LM Studio). Claude requires ANTHROPIC_API_KEY.")
+    args = ap.parse_args()
+
+    enrich_all(
+        in_path=Path(args.in_path),
+        out_path=Path(args.out),
+        cache_path=Path(args.cache),
+        limit=args.limit,
+        re_enrich=args.re_enrich,
+        only_failed=args.only_failed,
+        use_claude=(args.lm == "claude"),
+    )
+
+
+if __name__ == "__main__":
+    main()

enrichment.py

@@ -0,0 +1,266 @@
+"""
+enrichment.py — turn a Verse into an EnrichedVerse using the local LLM.
+
+This module is the heart of the redesign. Instead of hoping that vector
+similarity between a user's English question and a Sanskrit verse will find
+the right teaching, we run a one-time offline pass that asks the local LLM
+to translate each verse into the language a real person would use to seek
+help. The output gets stored alongside the verse and embedded for retrieval.
+
+What the prompt asks for, and why each field
+--------------------------------------------
+We extract six fields. Each one earns its place by closing a different gap
+between scripture and a user's question:
+
+  paraphrase             — what the verse teaches, in plain modern English.
+                           This is what the synthesizer reads when writing
+                           the advisor's reply, so paraphrase quality matters
+                           more than embedding quality.
+
+  themes                 — Vedānta concepts engaged. Tradition-native names
+                           (karma_yoga, vairagya, sakshi, two_truths). Used
+                           for filtering and for ensuring the metric can
+                           verify Advaita-coherence.
+
+  life_situations        — the predicaments where this verse helps. User-
+                           language. This is the field that does the actual
+                           bridging: a query about "facing failure" finds
+                           BG 2.47 even though those words aren't in the verse.
+
+  emotions_addressed     — drawn from a fixed vocabulary so we get faceted
+                           filtering rather than free-text drift. The metric
+                           uses this to verify that retrieved verses actually
+                           address the user's felt emotion.
+
+  practical_teaching     — what the verse asks the seeker to do or shift.
+                           The synthesizer uses this as the seed for its
+                           "concrete practice you can try this week" close.
+
+  hypothetical_questions — five questions a real person might bring to the
+                           verse. Highest-leverage field for retrieval recall.
+
+A closed vocabulary for emotions
+--------------------------------
+We constrain `emotions_addressed` to the EMOTION_VOCAB list below. If we let
+the LLM generate freely, we get drift: "sadness" / "sorrow" / "melancholy" /
+"grief-tinged blue" all become separate buckets, and faceted filtering
+becomes useless. Closed vocab keeps the index sharp.
+
+We don't constrain themes the same way because the Sanskrit conceptual
+vocabulary is open-ended and forcing the LLM into a small list would lose
+information. We just normalize for casing/spacing in post-processing.
+
+Working with a flaky local LLM
+------------------------------
+Local 26B-class models occasionally produce malformed structured output.
+This module assumes that. The enrich() function:
+  - validates output against minimum-quality checks
+  - retries up to 2 times with temperature=0
+  - on persistent failure, returns an EnrichedVerse with empty enrichment
+    fields rather than raising — so the corpus can still index on the
+    literal text + bhāṣya and the verse isn't lost
+"""
+
+from __future__ import annotations
+import re
+from dataclasses import asdict
+import dspy
+
+from corpus import Verse, EnrichedVerse
+
+
+# ──────────────────────────── Closed emotion vocabulary ────────────────────────────
+# Twenty buckets, ordered roughly from acute to diffuse. Adding entries is
+# easy; removing them risks orphaning previously-enriched records.
+EMOTION_VOCAB: tuple[str, ...] = (
+    "grief",                # acute loss
+    "anticipatory_grief",   # loss in advance
+    "fear",                 # discrete fear
+    "anxiety",              # chronic, diffuse
+    "despair",              # loss of hope
+    "shame",                # self-as-bad
+    "guilt",                # action-as-bad
+    "anger",
+    "resentment",
+    "envy",
+    "jealousy",
+    "longing",
+    "loneliness",
+    "doubt",                # epistemic; not knowing
+    "disillusionment",      # the hollowness of attained goals
+    "boredom",              # the inertness of repetition
+    "restlessness",         # the inability to settle
+    "frustration",
+    "confusion",
+    "numbness",             # affect-blunted
+)
+
+
+# ──────────────────────────── DSPy signature ────────────────────────────
+class EnrichVerse(dspy.Signature):
+    """You are an Advaita-Vedānta-trained reader producing structured metadata
+    for a verse from the Bhagavad Gītā or a related scripture, so that a
+    spiritual advisor can later find this verse when a real person describes
+    a life situation in everyday language. Stay strictly within the framework
+    of Śaṅkarācārya's non-dual interpretation. Do not import dualistic notions
+    (separate creator/creature, soul-merging-into-God-as-other, etc.) and do
+    not bypass the verse's plain meaning by always retreating to the absolute.
+
+    The verse may include the Sanskrit, the English translation, and (when
+    available) Śaṅkara's commentary. Read all three. Your output is structured
+    fields, not prose. Be specific, lived, concrete. Avoid generic spiritual
+    language ('find peace', 'be in the moment'). Avoid tradition-foreign
+    therapy language ('honor your feelings'). When in doubt about a field,
+    leave it shorter rather than padded."""
+
+    # Inputs — the verse in its richest available form
+    verse_ref: str = dspy.InputField(desc="Citation form, e.g. 'BG 2.47'.")
+    sanskrit: str = dspy.InputField(desc="Devanāgarī text, may be empty.")
+    translation: str = dspy.InputField(desc="English translation of the verse.")
+    bhashya: str = dspy.InputField(desc="Śaṅkara's commentary on this verse, may be empty.")
+
+    # Outputs
+    paraphrase: str = dspy.OutputField(
+        desc="One or two sentences in plain modern English stating what the "
+             "verse teaches. Not a translation; a teaching summary. No jargon."
+    )
+    themes: list[str] = dspy.OutputField(
+        desc="2–5 Vedānta concepts the verse engages, in tradition-native "
+             "vocabulary with snake_case_keys, e.g. ['karma_yoga', 'non_attachment', "
+             "'two_truths']. Use Sanskrit terms where they're the right name."
+    )
+    life_situations: list[str] = dspy.OutputField(
+        desc="3–6 specific human predicaments this verse would help with, "
+             "in everyday English. e.g. 'facing public failure after years of "
+             "effort'. NOT 'finding peace' or 'spiritual growth'."
+    )
+    emotions_addressed: list[str] = dspy.OutputField(
+        desc="The emotions this verse meets, drawn ONLY from this fixed list: "
+             + ", ".join(EMOTION_VOCAB) + ". 1–4 entries."
+    )
+    practical_teaching: str = dspy.OutputField(
+        desc="One sentence: what the verse asks the seeker to actually do or "
+             "shift. If the verse is purely ontological, write 'pure ontology — "
+             "no direct prescription' and the field will be ignored downstream."
+    )
+    hypothetical_questions: list[str] = dspy.OutputField(
+        desc="EXACTLY 5 first-person questions a real person might write to a "
+             "spiritual advisor that this verse would speak to. Specific, "
+             "ungeneric, in the user's voice. NOT in scripture's voice. e.g. "
+             "'I worked on this for three years and it just failed publicly — "
+             "how do I keep going?'"
+    )
+
+
+# ──────────────────────────── Validators ────────────────────────────
+THEME_KEY_RX = re.compile(r"^[a-z][a-z0-9_]{2,40}$")
+
+
+def _normalize_theme(t: str) -> str:
+    t = t.strip().lower()
+    t = re.sub(r"[\s\-]+", "_", t)
+    t = re.sub(r"[^a-z0-9_]", "", t)
+    return t
+
+
+def _validate(pred) -> tuple[bool, str]:
+    """Light schema check. Returns (ok, reason_if_not_ok). Used to decide
+    whether to retry the LM call with a stricter prompt."""
+    paraphrase = (pred.paraphrase or "").strip()
+    if len(paraphrase) < 20:
+        return False, "paraphrase too short"
+
+    qs = pred.hypothetical_questions or []
+    if not isinstance(qs, list) or len(qs) < 3:
+        return False, f"need ≥3 hypothetical_questions, got {len(qs)}"
+
+    sits = pred.life_situations or []
+    if not isinstance(sits, list) or len(sits) < 2:
+        return False, f"need ≥2 life_situations, got {len(sits)}"
+
+    emos = pred.emotions_addressed or []
+    if not isinstance(emos, list) or not emos:
+        return False, "emotions_addressed empty"
+    bad = [e for e in emos if _normalize_theme(e) not in EMOTION_VOCAB]
+    if bad:
+        return False, f"emotions outside vocabulary: {bad}"
+
+    themes = pred.themes or []
+    if not isinstance(themes, list) or not themes:
+        return False, "themes empty"
+
+    return True, ""
+
+
+# ──────────────────────────── Module ────────────────────────────
+class Enricher(dspy.Module):
+    """Wraps the EnrichVerse signature with retries and post-processing.
+
+    Why ChainOfThought over Predict
+    -------------------------------
+    GEPA may eventually optimize this prompt too, and ChainOfThought gives it
+    a `reasoning` trace to inspect during reflection. The cost is one extra
+    paragraph of LM output per call, which is negligible at our scale.
+    """
+
+    def __init__(self, max_retries: int = 2):
+        super().__init__()
+        self.predict = dspy.ChainOfThought(EnrichVerse)
+        self.max_retries = max_retries
+
+    def forward(self, verse: Verse) -> EnrichedVerse:
+        attempt = 0
+        last_err = ""
+        pred = None
+
+        while attempt <= self.max_retries:
+            try:
+                pred = self.predict(
+                    verse_ref=verse.verse_ref,
+                    sanskrit=verse.sanskrit or "",
+                    translation=verse.translation or "",
+                    bhashya=verse.bhashya or "",
+                )
+                ok, reason = _validate(pred)
+                if ok:
+                    break
+                last_err = reason
+            except Exception as e:
+                last_err = f"LM error: {e}"
+            attempt += 1
+
+        # Build the EnrichedVerse from the Verse + whatever we got
+        base = asdict(verse)
+        ev = EnrichedVerse(**base)
+
+        if pred and not last_err:
+            ev.paraphrase = (pred.paraphrase or "").strip()
+            ev.practical_teaching = (pred.practical_teaching or "").strip()
+            ev.themes = [
+                _normalize_theme(t) for t in (pred.themes or [])
+                if THEME_KEY_RX.match(_normalize_theme(t))
+            ]
+            ev.life_situations = [
+                s.strip() for s in (pred.life_situations or [])
+                if s and len(s.strip()) >= 5
+            ]
+            ev.emotions_addressed = [
+                _normalize_theme(e) for e in (pred.emotions_addressed or [])
+                if _normalize_theme(e) in EMOTION_VOCAB
+            ]
+            ev.hypothetical_questions = [
+                q.strip() for q in (pred.hypothetical_questions or [])
+                if q and len(q.strip()) >= 10
+            ][:5]  # cap at 5
+
+            # Stamp the model so re-runs after a model swap can be detected
+            try:
+                lm = dspy.settings.lm
+                ev.enrichment_model = getattr(lm, "model", "") or ""
+            except Exception:
+                pass
+        else:
+            # Enrichment failed; keep the verse but mark it
+            ev.enrichment_model = f"FAILED: {last_err}"
+
+        return ev

ingest_corpus.py

@@ -0,0 +1,203 @@
+"""
+ingest_corpus.py — run the parsers and produce data/corpus.jsonl.
+
+This script lives between download_sources.py (which gets bytes onto disk)
+and enrich_corpus.py (which adds LLM-derived fields). Its specific job:
+
+    1. Walk each enabled source in the registry.
+    2. Dispatch to its parser, which yields Verse records.
+    3. Merge records across sources by verse_ref.
+       - The Gītā parser yields verses with translation but no bhāṣya.
+       - The Sastry parser yields verses with bhāṣya but spotty translation.
+       - We want one record per verse, with both populated when possible.
+    4. Write the merged stream as JSONL to data/corpus.jsonl.
+
+Why merge by verse_ref rather than verse_id
+-------------------------------------------
+The Gītā parser uses work='bhagavad_gita' and the Sastry parser uses
+work='bhagavad_gita_bhashya'. Their verse_ids therefore differ (different
+work prefix), but their verse_refs match — both render as 'BG 2.47'. We
+key the merge on verse_ref since that's the reader-facing canonical citation.
+
+Conflict policy when merging
+----------------------------
+- Translation: keep whichever record has it; if both, prefer the one whose
+  source_key is in the GITA_TEXT_PRIORITY list. (We want the modern, clean
+  Sivananda over Sastry's archaic English-of-Śaṅkara-paraphrasing-the-verse.)
+- Bhāṣya: only one source produces this; conflicts shouldn't happen.
+- Sanskrit / transliteration / word_meanings: prefer gita_json; richer.
+"""
+
+from __future__ import annotations
+import argparse
+from collections import defaultdict
+from pathlib import Path
+from typing import Iterable
+
+from tqdm import tqdm
+
+import config
+from corpus import Verse, write_jsonl
+from sources_registry import enabled_sources, by_key, Source
+
+# Parsers
+from parsers import gita_json as parser_gita_json
+from parsers import sastry_archive as parser_sastry
+
+
+# When two sources both have a translation, this list decides which wins
+GITA_TEXT_PRIORITY = ("gita_json_core", "sastry_gita_bhashya")
+
+
+def _parse_source(src: Source, raw_dir: Path) -> Iterable[Verse]:
+    """Dispatch to the right parser for a registry entry.
+
+    Each parser is documented to take a directory and return an iterable of
+    Verses; this function is just a switch table.
+    """
+    if src.parser == "gita_json":
+        # The gita_json parser can take both the core dir and (optionally) a
+        # translations dir. We pass the same dir for both since the downloader
+        # puts all gita_json* files into per-source folders.
+        if src.key == "gita_json_core":
+            translations_dir = raw_dir.parent / "gita_json_translations"
+            return parser_gita_json.parse(
+                raw_dir,
+                translations_dir if translations_dir.exists() else None,
+            )
+        # The translations source is "consumed" alongside core, not parsed alone
+        return iter(())
+
+    if src.parser == "sastry_archive":
+        return parser_sastry.parse(raw_dir)
+
+    if src.parser == "wisdomlib_html":
+        # Stub for now — see parsers/wisdomlib_html.py to implement.
+        # We don't fail the whole ingest just because one parser is unimplemented.
+        print(f"[ingest] wisdomlib_html parser not implemented yet — skipping {src.key}")
+        return iter(())
+
+    if src.parser == "thibaut_sbe":
+        print(f"[ingest] thibaut_sbe parser not implemented yet — skipping {src.key}")
+        return iter(())
+
+    if src.parser == "plain_text":
+        # Reserved for user-dropped texts; future work
+        return iter(())
+
+    raise ValueError(f"Unknown parser type: {src.parser}")
+
+
+def _merge(records: list[Verse]) -> list[Verse]:
+    """Merge multiple parser outputs into one record per verse_ref.
+
+    The output preserves the order of first appearance, so the corpus.jsonl
+    file is naturally chapter-then-verse ordered.
+    """
+    by_ref: dict[str, Verse] = {}
+    order: list[str] = []
+
+    for r in records:
+        if r.verse_ref not in by_ref:
+            by_ref[r.verse_ref] = r
+            order.append(r.verse_ref)
+            continue
+
+        existing = by_ref[r.verse_ref]
+
+        # Translation: pick higher-priority source if both have one
+        new_translation = existing.translation
+        new_translator = existing.translator
+        if r.translation and (
+            not existing.translation
+            or _priority(r.source_key) < _priority(existing.source_key)
+        ):
+            new_translation = r.translation
+            new_translator = r.translator
+
+        # Bhashya: only one source typically has it, take whichever isn't blank
+        new_bhashya = existing.bhashya or r.bhashya
+        new_bhashya_tr = existing.bhashya_translator or r.bhashya_translator
+
+        # Sanskrit family of fields: prefer the existing record if it has them,
+        # else take from the new record
+        merged = Verse(
+            verse_id=existing.verse_id,
+            work=existing.work,           # keep the work_display of whichever came first
+            work_display=existing.work_display,
+            verse_ref=existing.verse_ref,
+            tier=_choose_tier(existing.tier, r.tier),
+            section=existing.section or r.section,
+            section_display=existing.section_display or r.section_display,
+            translation=new_translation,
+            translator=new_translator,
+            sanskrit=existing.sanskrit or r.sanskrit,
+            transliteration=existing.transliteration or r.transliteration,
+            word_meanings=existing.word_meanings or r.word_meanings,
+            bhashya=new_bhashya,
+            bhashya_translator=new_bhashya_tr,
+            source_key=existing.source_key + "+" + r.source_key,
+            license=existing.license or r.license,
+        )
+        by_ref[r.verse_ref] = merged
+
+    return [by_ref[k] for k in order]
+
+
+def _priority(source_key: str) -> int:
+    """Lower is higher-priority. Sources not in the priority list rank last."""
+    for i, key in enumerate(GITA_TEXT_PRIORITY):
+        if source_key == key or source_key.startswith(key + "+") or source_key.endswith("+" + key):
+            return i
+    return 99
+
+
+def _choose_tier(a: str, b: str) -> str:
+    """When two records merge, the tier of the merged verse is the most
+    'authoritative' of the two: primary > shankara > supporting.
+
+    Why primary > shankara: when we have both the verse text (primary) and
+    Śaṅkara's bhāṣya on it (shankara) folded into one record, the verse
+    itself is what the citation refers to — so primary wins."""
+    rank = {"primary": 0, "shankara": 1, "supporting": 2}
+    return a if rank.get(a, 9) <= rank.get(b, 9) else b
+
+
+# ──────────────────────────── CLI ────────────────────────────
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--out", default=str(config.DATA_DIR / "corpus.jsonl"))
+    args = ap.parse_args()
+
+    raw_root = config.DATA_DIR / "raw"
+    if not raw_root.exists():
+        raise SystemExit("data/raw/ doesn't exist. Run download_sources.py first.")
+
+    all_records: list[Verse] = []
+    for src in enabled_sources():
+        raw_dir = raw_root / src.key
+        if not raw_dir.exists():
+            print(f"[ingest] {src.key}: no files at {raw_dir}; skipping")
+            continue
+        print(f"[ingest] parsing {src.key} via {src.parser}")
+        try:
+            n_before = len(all_records)
+            for v in _parse_source(src, raw_dir):
+                if v.has_content():
+                    all_records.append(v)
+            print(f"[ingest]   yielded {len(all_records) - n_before} records")
+        except Exception as e:
+            print(f"[ingest] {src.key} failed: {e}")
+
+    print(f"[ingest] merging {len(all_records)} records by verse_ref ...")
+    merged = _merge(all_records)
+    print(f"[ingest] {len(merged)} unique verses after merge")
+
+    out_path = Path(args.out)
+    n = write_jsonl(merged, out_path)
+    print(f"[ingest] wrote {n} verses to {out_path}")
+    print(f"[ingest] next: python enrich_corpus.py")
+
+
+if __name__ == "__main__":
+    main()

knowledge_base.py

@@ -0,0 +1,416 @@
+"""
+knowledge_base.py — verse-indexed, multi-view RAG over the enriched corpus.
+
+The shift from the old design
+-----------------------------
+The old knowledge_base.py chunked source text into 380-token windows with
+overlap. The new one indexes each verse as a single record but with three
+*views* — three separate embeddings of three different framings of the same
+verse — so that queries phrased in different registers can all find it.
+
+The three views per verse, and what each one is good for:
+
+    literal_view   — the English translation (and Sanskrit fragment if
+                     available). Best for queries that share lexical features
+                     with the text itself: "what does it mean to act without
+                     attachment?" maps cleanly to BG 2.47's literal text.
+
+    bhashya_view   — Śaṅkara's commentary on the verse. Best for queries that
+                     ask about the Vedāntic explanation rather than the verse
+                     itself: "how does adhyāsa relate to suffering?" finds
+                     the bhāṣya passages where Śaṅkara unfolds adhyāsa.
+
+    advisor_view   — the LLM-enriched composite (paraphrase + life situations
+                     + emotions addressed + hypothetical questions). Best for
+                     real-world questions in real-world language. This is
+                     where the language gap closes.
+
+At retrieval time we query all three indices, merge by verse_id (so each
+verse appears at most once), and combine scores with a weighted sum that
+gives the advisor_view the lion's share of credit while letting the
+literal and bhāṣya views catch cases the LLM enrichment missed.
+
+Why three indices and not one with concatenated views
+-----------------------------------------------------
+Concatenating literal + bhāṣya + advisor into one big text and embedding
+that gives you the average direction across the three. Real semantic search
+benefits from being able to match any one of the three angles strongly. The
+extra storage (three vectors per verse instead of one) is trivial; the
+retrieval-quality difference is large.
+
+Storage layout
+--------------
+We keep three Chroma collections in artifacts/chroma/:
+    advaita_literal
+    advaita_bhashya
+    advaita_advisor
+
+Each holds the same set of verse_ids. We resolve a hit's full record by
+reading data/corpus_enriched.jsonl (kept small enough to live in memory).
+"""
+
+from __future__ import annotations
+import argparse
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Iterable
+
+import chromadb
+from chromadb.config import Settings
+from sentence_transformers import SentenceTransformer
+from tqdm import tqdm
+
+import config
+from corpus import EnrichedVerse, read_jsonl_enriched
+
+
+# ──────────────────────────── Constants ────────────────────────────
+COLLECTION_LITERAL = "advaita_literal"
+COLLECTION_BHASHYA = "advaita_bhashya"
+COLLECTION_ADVISOR = "advaita_advisor"
+
+# Tier weights — multiplied into the cosine similarity at retrieval time.
+# Same logic as before: primary scripture and Śaṅkara's pen outrank later
+# voices when the cosine score is otherwise comparable.
+TIER_WEIGHTS = {"primary": 1.10, "shankara": 1.10, "supporting": 1.00}
+
+# View weights — how much each view's score contributes to the combined
+# score per verse. The advisor view dominates because it is the one
+# designed to bridge the language gap; literal and bhāṣya are insurance
+# against the enrichment pipeline missing a topic.
+VIEW_WEIGHTS = {"advisor": 0.55, "literal": 0.25, "bhashya": 0.20}
+
+
+# ──────────────────────────── Hit dataclass ────────────────────────────
+@dataclass
+class Hit:
+    """One retrieval result, post-merge across the three views."""
+    verse: EnrichedVerse
+    combined_score: float                                       # used for ranking
+    view_scores: dict[str, float] = field(default_factory=dict) # diagnostics
+
+    def __repr__(self) -> str:
+        v = self.verse
+        return (f"Hit({v.verse_ref}, tier={v.tier}, "
+                f"score={self.combined_score:.3f}, views={self.view_scores})")
+
+    def to_dict(self) -> dict:
+        """Flatten a Hit to a JSON-serializable dict so the advisor can carry
+        it in dspy.Prediction (which is pickled during GEPA optimization), and
+        so the metric can read its fields without importing this module."""
+        v = self.verse
+        return {
+            "verse_id": v.verse_id,
+            "verse_ref": v.verse_ref,
+            "work": v.work,
+            "work_display": v.work_display,
+            "section": v.section,
+            "tier": v.tier,
+            "translation": v.translation,
+            "translator": v.translator,
+            "bhashya": v.bhashya,
+            "bhashya_translator": v.bhashya_translator,
+            "paraphrase": v.paraphrase,
+            "themes": list(v.themes),
+            "life_situations": list(v.life_situations),
+            "emotions_addressed": list(v.emotions_addressed),
+            "hypothetical_questions": list(v.hypothetical_questions),
+            "score": self.combined_score,
+            "view_scores": dict(self.view_scores),
+            # Legacy alias the old metric used:
+            "meta": {
+                "verse_ref": v.verse_ref,
+                "work": v.work,
+                "section": v.section,
+                "tier": v.tier,
+            },
+        }
+
+
+# ──────────────────────────── Internals ────────────────────────────
+def _client() -> chromadb.api.ClientAPI:
+    return chromadb.PersistentClient(
+        path=str(config.CHROMA_DIR),
+        settings=Settings(anonymized_telemetry=False),
+    )
+
+
+def _embedder() -> SentenceTransformer:
+    return SentenceTransformer(config.EMBED_MODEL, device=config.EMBED_DEVICE)
+
+
+def _record_metadata(v: EnrichedVerse) -> dict:
+    """Metadata stored alongside each chroma record so the retriever can
+    filter and report without re-loading the JSONL on every call.
+
+    chromadb requires scalar metadata values, so list-valued fields (themes,
+    emotions) are joined with semicolons. The choice of ';' is safe because
+    neither chroma nor our snake_case theme keys contain that character.
+    """
+    return {
+        "verse_id": v.verse_id,
+        "verse_ref": v.verse_ref,
+        "work": v.work,
+        "tier": v.tier,
+        "section": v.section,
+        "themes_csv": ";".join(v.themes),
+        "emotions_csv": ";".join(v.emotions_addressed),
+    }
+
+
+# ──────────────────────────── Index build ────────────────────────────
+def build_index(corpus_path: Path | None = None) -> dict[str, int]:
+    """(Re)build all three view-indices from the enriched corpus.
+
+    Returns a dict {view_name: n_records} for confirmation. The function is
+    safe to re-run; it deletes existing collections first so partial state
+    from a prior crash doesn't pollute results.
+    """
+    corpus_path = corpus_path or (config.DATA_DIR / "corpus_enriched.jsonl")
+    if not corpus_path.exists():
+        raise SystemExit(
+            f"No enriched corpus at {corpus_path}.\n"
+            f"Pipeline: download_sources.py → ingest_corpus.py → "
+            f"enrich_corpus.py → knowledge_base.py --build"
+        )
+
+    print(f"Loading embedding model: {config.EMBED_MODEL} on {config.EMBED_DEVICE}")
+    embedder = _embedder()
+
+    client = _client()
+    # Drop existing collections; build_index is "rebuild from scratch"
+    for name in (COLLECTION_LITERAL, COLLECTION_BHASHYA, COLLECTION_ADVISOR):
+        try:
+            client.delete_collection(name)
+        except Exception:
+            pass
+
+    coll_literal = client.create_collection(
+        COLLECTION_LITERAL, metadata={"hnsw:space": "cosine"})
+    coll_bhashya = client.create_collection(
+        COLLECTION_BHASHYA, metadata={"hnsw:space": "cosine"})
+    coll_advisor = client.create_collection(
+        COLLECTION_ADVISOR, metadata={"hnsw:space": "cosine"})
+
+    verses = list(read_jsonl_enriched(corpus_path))
+    print(f"Indexing {len(verses)} verses across 3 views ...")
+
+    counts = {"literal": 0, "bhashya": 0, "advisor": 0}
+
+    # We batch by view so each call to encode() is efficient. For 3000 verses
+    # at small-batch BGE this is a few seconds total per view, much faster
+    # than one-at-a-time embedding.
+    BATCH = 64
+    for view_name, view_fn, coll in (
+        ("literal", lambda v: v.literal_view(), coll_literal),
+        ("bhashya", lambda v: v.bhashya_view(), coll_bhashya),
+        ("advisor", lambda v: v.advisor_view(), coll_advisor),
+    ):
+        # Skip verses whose view is empty. A verse without a bhāṣya simply
+        # doesn't appear in the bhāṣya index — the merger handles partial
+        # coverage cleanly.
+        records = [(v, view_fn(v)) for v in verses]
+        records = [(v, t) for v, t in records if t.strip()]
+
+        for i in tqdm(range(0, len(records), BATCH), desc=f"  view: {view_name}"):
+            chunk = records[i:i + BATCH]
+            ids = [v.verse_id for v, _ in chunk]
+            texts = [t for _, t in chunk]
+            metas = [_record_metadata(v) for v, _ in chunk]
+
+            vectors = embedder.encode(
+                texts,
+                normalize_embeddings=True,
+                show_progress_bar=False,
+                batch_size=BATCH,
+            ).tolist()
+            coll.add(ids=ids, embeddings=vectors,
+                     documents=texts, metadatas=metas)
+
+        counts[view_name] = len(records)
+
+    print(f"Index built: {counts}")
+    return counts
+
+
+# ───────���──────────────────── Retriever ────────────────────────────
+class AdvaitaRetriever:
+    """Multi-view retriever returning Hit objects backed by EnrichedVerse.
+
+    Construction loads the enriched corpus into memory (≈3000 records,
+    ≈10 MB) so we can resolve hits to full records without per-call disk
+    reads. This matters during GEPA optimization, which calls retrieve()
+    hundreds of times per evaluation pass.
+
+    The retriever is intentionally light: it doesn't filter by metadata,
+    by tier, or by emotion at query time. Filtering happens at scoring
+    (TIER_WEIGHTS) and at the SelectPassages stage downstream. Keeping
+    retrieval permissive and selection picky is more robust than the
+    reverse — when retrieval over-filters, you can never recover the
+    missed verse later in the pipeline.
+    """
+
+    def __init__(self, top_k: int = config.TOP_K_RETRIEVE,
+                 corpus_path: Path | None = None):
+        self.top_k = top_k
+        self._embedder: SentenceTransformer | None = None
+        self._coll_literal = None
+        self._coll_bhashya = None
+        self._coll_advisor = None
+
+        cp = corpus_path or (config.DATA_DIR / "corpus_enriched.jsonl")
+        self._verses_by_id: dict[str, EnrichedVerse] = {
+            v.verse_id: v for v in read_jsonl_enriched(cp)
+        }
+
+    def _ensure(self):
+        """Lazy-load embedder and collections. We avoid loading at __init__
+        so a process that only needs the corpus mapping (e.g. the metric)
+        doesn't pay the SentenceTransformer load time."""
+        if self._embedder is None:
+            self._embedder = _embedder()
+        if self._coll_advisor is None:
+            client = _client()
+            self._coll_literal = client.get_collection(COLLECTION_LITERAL)
+            self._coll_bhashya = client.get_collection(COLLECTION_BHASHYA)
+            self._coll_advisor = client.get_collection(COLLECTION_ADVISOR)
+
+    def search(self, query: str, k: int | None = None) -> list[Hit]:
+        """Run the query against all three views, merge by verse_id, and
+        return the top-k Hits sorted by combined score."""
+        self._ensure()
+        k = k or self.top_k
+
+        q_emb = self._embedder.encode(
+            [query], normalize_embeddings=True, show_progress_bar=False
+        ).tolist()
+
+        # Over-fetch from each view; we want enough overlap that the merge
+        # has something to work with. 3*k per view is a reasonable upper
+        # bound: large enough to catch verses one view ranked low and another
+        # ranked high, small enough that Chroma's HNSW stays fast.
+        per_view_k = max(8, k * 3)
+
+        view_results: dict[str, list[tuple[str, float, dict]]] = {}
+        for name, coll in (("literal", self._coll_literal),
+                           ("bhashya", self._coll_bhashya),
+                           ("advisor", self._coll_advisor)):
+            r = coll.query(query_embeddings=q_emb, n_results=per_view_k)
+            ids = r["ids"][0]
+            dists = r["distances"][0]   # cosine distance, in [0, 2]
+            metas = r["metadatas"][0]
+            view_results[name] = list(zip(ids, dists, metas))
+
+        # Merge: for each verse_id seen in any view, compute its combined
+        # score as Σ_v VIEW_WEIGHTS[v] * cos_sim(v) * tier_weight, where any
+        # view that didn't return that verse contributes 0. This is a soft
+        # voting scheme: a verse that appears strongly in one view but not
+        # others can still rank highly if that one view's weight is enough.
+        per_verse: dict[str, dict[str, float]] = {}
+        per_verse_meta: dict[str, dict] = {}
+        for view_name, results in view_results.items():
+            for vid, dist, meta in results:
+                cos_sim = 1.0 - dist
+                per_verse.setdefault(vid, {})[view_name] = cos_sim
+                per_verse_meta[vid] = meta
+
+        hits: list[Hit] = []
+        for vid, view_scores in per_verse.items():
+            tier = per_verse_meta[vid].get("tier", "supporting")
+            tw = TIER_WEIGHTS.get(tier, 1.0)
+            combined = sum(
+                VIEW_WEIGHTS[v] * view_scores.get(v, 0.0)
+                for v in VIEW_WEIGHTS
+            ) * tw
+            verse = self._verses_by_id.get(vid)
+            if verse is None:
+                # Index has it but corpus file doesn't — corpus and index
+                # have drifted. Skip rather than fabricate a record.
+                continue
+            hits.append(Hit(verse=verse,
+                            combined_score=combined,
+                            view_scores=view_scores))
+
+        hits.sort(key=lambda h: h.combined_score, reverse=True)
+        return hits[:k]
+
+    def search_many(self, queries: Iterable[str],
+                    k_per: int | None = None) -> list[Hit]:
+        """Run multiple queries (e.g. from PlanRetrieval) and dedupe by
+        verse_id, keeping the highest combined score across queries."""
+        seen: dict[str, Hit] = {}
+        for q in queries:
+            for h in self.search(q, k=k_per):
+                cur = seen.get(h.verse.verse_id)
+                if cur is None or h.combined_score > cur.combined_score:
+                    seen[h.verse.verse_id] = h
+        out = list(seen.values())
+        out.sort(key=lambda h: h.combined_score, reverse=True)
+        return out
+
+
+# ──────────────────────────── Formatter for the LLM ────────────────────────────
+def format_hits_for_llm(hits: list[Hit]) -> str:
+    """Render hits for the SelectPassages and SynthesizeAdvice prompts.
+
+    We expose the verse_ref (so the synthesizer can cite it), the literal
+    translation (so the synthesizer can quote it lightly), the bhāṣya
+    snippet (so the synthesizer can ground its claims), and the advisor-view
+    fields (so the synthesizer knows *why* this verse is being suggested for
+    this user).
+
+    Each hit is bounded in length so the prompt stays tractable on a 26B
+    local model with an 8k context window.
+    """
+    blocks = []
+    for i, h in enumerate(hits, start=1):
+        v = h.verse
+        block = [f"[{i}] {v.verse_ref} — {v.work_display}, {v.section_display}"]
+        block.append(f"    tier: {v.tier}   score: {h.combined_score:.3f}")
+        if v.translation:
+            block.append(f"    Translation: {v.translation.strip()[:600]}")
+        if v.bhashya:
+            block.append(f"    Bhāṣya (Śaṅkara): {v.bhashya.strip()[:800]}")
+        if v.paraphrase:
+            block.append(f"    Teaching: {v.paraphrase}")
+        if v.life_situations:
+            block.append(f"    Speaks to: {'; '.join(v.life_situations)}")
+        if v.emotions_addressed:
+            block.append(f"    Addresses: {', '.join(v.emotions_addressed)}")
+        if v.themes:
+            block.append(f"    Themes: {', '.join(v.themes)}")
+        blocks.append("\n".join(block))
+    return "\n\n".join(blocks)
+
+
+# Alias kept so that advisor.py — and any prior code that imported the old
+# name — works without modification. Both names refer to the same function
+# because the new "passages" the advisor sees ARE Hit objects backed by
+# EnrichedVerse records; the rendering is identical.
+format_passages_for_llm = format_hits_for_llm
+
+
+# ──────────────────────────── CLI ────────────────────────────
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--build", action="store_true",
+                    help="(Re)build the multi-view index from corpus_enriched.jsonl")
+    ap.add_argument("--query", type=str, default=None,
+                    help="Run a test query against the index")
+    args = ap.parse_args()
+
+    if args.build:
+        build_index()
+        return
+
+    if args.query:
+        retr = AdvaitaRetriever()
+        hits = retr.search(args.query)
+        print(format_hits_for_llm(hits))
+        return
+
+    ap.print_help()
+
+
+if __name__ == "__main__":
+    main()

metrics.py

@@ -0,0 +1,435 @@
+"""
+metrics.py — the metric is the specification.
+
+GEPA optimizes whatever the metric rewards. So the metric here is not a single
+number; it's a *contract* on what an Advaita-grounded, empathetic, practically
+useful response looks like — combined with rich textual feedback the reflection
+LM uses to rewrite prompts.
+
+We combine three signals:
+    1. Rule-based checks (fast, deterministic)
+        - citation grounding (cites real retrieved sources, not hallucinated)
+        - tier preference (primary + Śaṅkara > supporting)
+        - structural hygiene (length, has actionable element, no therapy clichés)
+    2. LLM-as-judge rubric scoring
+        - Advaita coherence (non-dual, not crypto-dualist)
+        - two-truths discipline (vyāvahārika ↔ pāramārthika)
+        - empathy without dissolving into the user's frame
+        - wit calibration (light around the predicament, never the pain)
+    3. Composite score + structured feedback string
+
+The function signature matches GEPA's metric contract:
+    metric(gold, pred, trace=None, pred_name=None, pred_trace=None) -> dspy.Prediction
+
+Returning dspy.Prediction(score=float, feedback=str) is the GEPA happy path.
+"""
+
+from __future__ import annotations
+import re
+import json
+from typing import Any
+import dspy
+
+
+# ──────────────────────────── Rule-based checks ────────────────────────────
+THERAPY_CLICHES = [
+    "you got this",
+    "be kind to yourself",
+    "self-care",
+    "just remember",
+    "trust the process",
+    "everything happens for a reason",
+    "you are enough",
+    "love and light",
+    "manifesting",
+    "send positive vibes",
+    "good vibes",
+]
+
+# Loose pattern catching citations like "BG 2.47", "Gītā 18.66", "Bṛhadāraṇyaka 4.4.5",
+# "Vivekacūḍāmaṇi 11", "Kaṭha Up. 1.3.14", etc.
+CITATION_PATTERN = re.compile(
+    r"\b("
+    r"BG\s*\d+[\.:]\d+"                                  # BG 2.47
+    r"|G[īi]t[āa]\s*\d+[\.:]\d+"                         # Gita 2.47
+    r"|[A-ZĀĪŪṚḌṬṆṢŚḤṂa-zāīūṛḍṭṇṣśḥṃ]{3,}\s*Up\.?\s*\d+(?:[\.:]\d+){0,2}"  # Kaṭha Up. 1.2.3
+    r"|Vivekac[ūu]ḍāmaṇi\s*\d+"
+    r"|Ātmabodha\s*\d+"
+    r"|Tattvabodha\s*\d+"
+    r"|Brahma\s*S[ūu]tra\s*\d+[\.:]\d+(?:[\.:]\d+)?"
+    r"|Aṣṭāvakra\s*G[īi]t[āa]\s*\d+[\.:]\d+"
+    r")\b"
+)
+
+EMPATHY_OPENERS = [
+    "what you", "you're carrying", "you are carrying", "i hear",
+    "this hurts", "this is painful", "the weight", "sitting with",
+    "what you describe", "the ache",
+]
+
+ACTIONABLE_MARKERS = [
+    "this week", "today", "try this", "begin by", "for the next",
+    "each morning", "each evening", "when you notice", "the next time",
+    "as a practice", "sit for", "spend ", "over the next",
+]
+
+NON_DUAL_MARKERS = [
+    "witness", "sākṣī", "sakshi", "non-dual", "advaita",
+    "pāramārthika", "paramarthika", "vyāvahārika", "vyavaharika",
+    "ātman", "atman", "brahman", "adhyāsa", "adhyasa", "māyā", "maya",
+    "neti neti", "tat tvam asi", "ahaṁ brahmāsmi", "aham brahmasmi",
+    "self with a capital", "the seer", "awareness itself",
+]
+
+
+def _word_count(s: str) -> int:
+    return len(s.split())
+
+
+def _has_any(text: str, needles: list[str]) -> list[str]:
+    low = text.lower()
+    return [n for n in needles if n in low]
+
+
+def _normalize_for_match(s: str) -> str:
+    return re.sub(r"\s+", " ", s.lower()).strip()
+
+
+def _citation_grounding(
+    sources_cited: list[str],
+    retrieved_passages: list[dict],
+) -> tuple[float, list[str], list[str]]:
+    """Return (grounding_score, grounded_citations, ungrounded_citations).
+
+    With the verse-indexed corpus, each retrieved passage carries an exact
+    verse_ref string ('BG 2.47', 'Muṇḍaka Up. 2.1.3', etc.). Grounding becomes
+    an exact set-membership test rather than fuzzy substring matching, which
+    is dramatically sharper feedback for GEPA's reflection step: 'BG 2.47'
+    is grounded if and only if 'BG 2.47' was in the retrieved set.
+
+    We still tolerate light formatting noise: the synthesizer might write
+    'BG 2.47', 'Bhagavad Gītā 2.47', 'Gita 2:47', etc. We canonicalize to
+    'BG <chap>.<verse>' for Gītā citations before comparing. Other works
+    are matched directly by verse_ref string with whitespace normalized.
+    """
+    if not sources_cited:
+        return 0.0, [], []
+
+    retrieved_refs = {
+        _canonicalize_ref(h.get("verse_ref") or h.get("meta", {}).get("verse_ref", ""))
+        for h in retrieved_passages
+    }
+    retrieved_refs.discard("")
+
+    grounded, ungrounded = [], []
+    for c in sources_cited:
+        canon = _canonicalize_ref(c)
+        # Try direct match first, then a "substring of any retrieved" fallback
+        # for cases where the synthesizer paraphrases the citation
+        # ('chapter 2 verse 47' vs 'BG 2.47').
+        hit = canon in retrieved_refs or any(
+            canon and (canon in r or r in canon) for r in retrieved_refs
+        )
+        (grounded if hit else ungrounded).append(c)
+
+    score = len(grounded) / max(len(sources_cited), 1)
+    return score, grounded, ungrounded
+
+
+def _canonicalize_ref(s: str) -> str:
+    """Normalize a citation string so 'BG 2.47', 'Bhagavad Gītā 2.47',
+    'Gītā 2:47' all reduce to the same canonical form 'BG 2.47'."""
+    s = re.sub(r"\s+", " ", s.strip())
+    # Gītā variants
+    m = re.match(r"^(?:BG|Bhagavad\s*G[īi]t[āa]|G[īi]t[āa])\s*(\d+)[\.:](\d+)", s, re.I)
+    if m:
+        return f"BG {int(m.group(1))}.{int(m.group(2))}"
+    # Default: lowercased, colons → dots
+    return s.lower().replace(":", ".")
+
+
+def _tier_preference(
+    sources_cited: list[str],
+    retrieved_passages: list[dict],
+    selected_indices: list[int],
+) -> tuple[float, dict]:
+    """Reward responses whose *cited* passages came from primary/Śaṅkara tiers."""
+    if not selected_indices:
+        return 0.0, {"primary": 0, "shankara": 0, "supporting": 0}
+
+    counts = {"primary": 0, "shankara": 0, "supporting": 0}
+    for idx in selected_indices:
+        if 1 <= idx <= len(retrieved_passages):
+            tier = retrieved_passages[idx - 1].get("meta", {}).get("tier", "supporting")
+            counts[tier] = counts.get(tier, 0) + 1
+
+    total = sum(counts.values()) or 1
+    preferred = counts["primary"] + counts["shankara"]
+    return preferred / total, counts
+
+
+def rule_based_score(pred: dspy.Prediction) -> tuple[float, dict]:
+    """Returns (score in [0,1], breakdown dict)."""
+    response = getattr(pred, "response", "") or ""
+    sources_cited = getattr(pred, "sources_cited", []) or []
+    retrieved = getattr(pred, "retrieved_passages", []) or []
+    selected_idx = getattr(pred, "selected_indices", []) or []
+    felt = getattr(pred, "felt_emotion", "") or ""
+
+    wc = _word_count(response)
+    length_ok = 200 <= wc <= 600
+    length_score = 1.0 if length_ok else max(0.0, 1.0 - abs(wc - 350) / 350)
+
+    citations_in_text = CITATION_PATTERN.findall(response)
+    has_citation = bool(citations_in_text) or bool(sources_cited)
+    citation_score = 1.0 if has_citation else 0.0
+
+    grounding_score, grounded, ungrounded = _citation_grounding(sources_cited, retrieved)
+
+    tier_score, tier_counts = _tier_preference(sources_cited, retrieved, selected_idx)
+
+    cliches = _has_any(response, THERAPY_CLICHES)
+    cliche_penalty = min(1.0, 0.25 * len(cliches))
+    cliche_score = 1.0 - cliche_penalty
+
+    # Empathy: opening should signal acknowledgement of feeling
+    head = response[:300].lower()
+    empathy_hits = [m for m in EMPATHY_OPENERS if m in head]
+    # Bonus if the felt_emotion content is referenced (loosely)
+    if felt:
+        for tok in felt.lower().split():
+            if len(tok) > 4 and tok in head:
+                empathy_hits.append(f"echoes:{tok}")
+                break
+    empathy_score = min(1.0, 0.4 + 0.3 * len(empathy_hits))
+
+    actionable_hits = _has_any(response, ACTIONABLE_MARKERS)
+    actionable_score = 1.0 if actionable_hits else 0.4
+
+    nondual_hits = _has_any(response, NON_DUAL_MARKERS)
+    nondual_score = min(1.0, 0.4 + 0.2 * len(nondual_hits))
+
+    # Weighted aggregate
+    components = {
+        "length": (length_score, 0.05),
+        "citation_present": (citation_score, 0.08),
+        "citation_grounding": (grounding_score, 0.18),
+        "tier_preference": (tier_score, 0.12),
+        "no_cliches": (cliche_score, 0.10),
+        "empathy_opening": (empathy_score, 0.15),
+        "actionable": (actionable_score, 0.10),
+        "nondual_register": (nondual_score, 0.22),
+    }
+    score = sum(s * w for s, w in components.values())
+
+    breakdown = {
+        "score": score,
+        "word_count": wc,
+        "components": {k: round(v[0], 3) for k, v in components.items()},
+        "citations_in_text": citations_in_text,
+        "sources_cited": sources_cited,
+        "grounded_citations": grounded,
+        "ungrounded_citations": ungrounded,
+        "tier_counts": tier_counts,
+        "therapy_cliches_found": cliches,
+        "empathy_hits": empathy_hits,
+        "actionable_hits": actionable_hits,
+        "nondual_markers_found": nondual_hits,
+    }
+    return score, breakdown
+
+
+# ──────────────────────────── LLM-judge rubric ────────────────────────────
+class JudgeAdvice(dspy.Signature):
+    """You are an examiner of Advaita-Vedānta spiritual counsel in the lineage
+    of Ādi Śaṅkarācārya. Score the advisor's response against the user's
+    question on each rubric (0.0 to 1.0) and write a short critique that an
+    optimizer can use to *improve the prompts that produced this response*.
+
+    Rubrics:
+
+    - advaita_coherence: Does the response reflect genuine non-dualism
+      (jīva-ātman-brahman identity), or does it accidentally smuggle in dualism
+      ('the soul reaches God', 'becoming one with the universe' as if they were
+      separate, etc.)? Does it avoid collapsing into nihilism ('nothing is
+      real')?
+
+    - two_truths_discipline: Does it honor the distinction between
+      vyāvahārika (transactional, where the user's pain and choices are real
+      and matter) and pāramārthika (absolute, where the witness is untouched)?
+      Failure modes: spiritual bypass (denying the pain by pointing to the
+      absolute), or pure-therapy register (forgetting the absolute exists).
+
+    - empathy_without_dissolving: Does it meet the user in their felt
+      experience without either flattening into therapy-speak OR dismissing
+      the feeling with premature transcendence?
+
+    - wit_calibration: Is there a light, dry touch around the cosmic
+      predicament (Śaṅkara himself is dry; this is consistent with the
+      tradition) WITHOUT being flippant about the user's actual pain? Both
+      'too solemn throughout' and 'making jokes about their situation' lose
+      points.
+
+    - source_integration: Are scriptural citations woven into the prose
+      (illuminating the point) rather than dumped as block quotes or used
+      as decoration? Are the references specific (Gītā 2.47, not just
+      "the Gita says")?
+
+    - practical_offering: Does the response close with something the user
+      can actually try — a question to sit with, a practice, a perspective
+      shift — rather than abstract platitudes?
+
+    - draw_from_personal_experiences: Does the response use parables and day to day life
+      stories as examples to encourage the user to relate better to the advise
+    
+    The critique should be specific and prescriptive: what to keep, what to
+    cut, what's missing. Phrase it as you would to a writer revising a draft."""
+
+    user_question: str = dspy.InputField()
+    response: str = dspy.InputField()
+    sources_cited: list[str] = dspy.InputField()
+
+    advaita_coherence: float = dspy.OutputField(desc="0.0 to 1.0")
+    two_truths_discipline: float = dspy.OutputField(desc="0.0 to 1.0")
+    empathy_without_dissolving: float = dspy.OutputField(desc="0.0 to 1.0")
+    wit_calibration: float = dspy.OutputField(desc="0.0 to 1.0")
+    source_integration: float = dspy.OutputField(desc="0.0 to 1.0")
+    practical_offering: float = dspy.OutputField(desc="0.0 to 1.0")
+    draw_from_personal_experiences: float = dspy.OutputField(desc="0.0 to 1.0")
+    critique: str = dspy.OutputField(
+        desc="3-6 sentences of prescriptive feedback for revising the response."
+    )
+
+
+# Lazily-instantiated judge. Call configure_judge() to use a stronger LM (e.g. gpt-4o)
+# during GEPA optimization so the reflection LM gets high-quality signal to work from.
+_judge = None
+_judge_lm = None  # None means use the globally-configured LM (task LM)
+
+
+def configure_judge(lm) -> None:
+    """Set the LM used by judge_score. Call before GEPA to use gpt-4o instead of the task LM."""
+    global _judge_lm, _judge
+    _judge_lm = lm
+    _judge = None  # reset so next call recreates with new context
+
+
+def _get_judge():
+    global _judge
+    if _judge is None:
+        _judge = dspy.ChainOfThought(JudgeAdvice)
+    return _judge
+
+
+def judge_score(user_question: str, pred: dspy.Prediction) -> tuple[float, dict, str]:
+    judge = _get_judge()
+    try:
+        call_kwargs = dict(
+            user_question=user_question,
+            response=getattr(pred, "response", "") or "",
+            sources_cited=getattr(pred, "sources_cited", []) or [],
+        )
+        if _judge_lm is not None:
+            with dspy.context(lm=_judge_lm):
+                j = judge(**call_kwargs)
+        else:
+            j = judge(**call_kwargs)
+    except Exception as e:
+        # If the judge fails (parse error, LM hiccup), fall back gracefully.
+        return 0.5, {"judge_error": str(e)}, f"Judge failed: {e}"
+
+    rubric = {
+        "advaita_coherence": float(j.advaita_coherence or 0.0),
+        "two_truths_discipline": float(j.two_truths_discipline or 0.0),
+        "empathy_without_dissolving": float(j.empathy_without_dissolving or 0.0),
+        "wit_calibration": float(j.wit_calibration or 0.0),
+        "source_integration": float(j.source_integration or 0.0),
+        "practical_offering": float(j.practical_offering or 0.0),
+        "draw_from_personal_experiences": float(j.draw_from_personal_experiences or 0.0),
+    }
+    weights = {
+        "advaita_coherence": 0.25,
+        "two_truths_discipline": 0.20,
+        "empathy_without_dissolving": 0.20,
+        "wit_calibration": 0.10,
+        "source_integration": 0.10,
+        "practical_offering": 0.10,
+        "draw_from_personal_experiences": 0.05,
+    }
+    score = sum(rubric[k] * weights[k] for k in rubric)
+    score = max(0.0, min(1.0, score))
+    return score, rubric, j.critique or ""
+
+
+# ──────────────────────────── Composite GEPA metric ────────────────────────────
+RULE_WEIGHT = 0.45
+JUDGE_WEIGHT = 0.55
+
+
+def _format_feedback(rule_breakdown: dict, judge_rubric: dict, critique: str) -> str:
+    """Concatenate rule-based facts and judge critique into one feedback string
+    that the GEPA reflection LM can read and use to rewrite prompts."""
+    lines = ["FEEDBACK FOR PROMPT IMPROVEMENT", ""]
+
+    lines.append("Rule-based observations:")
+    comps = rule_breakdown.get("components", {})
+    for k, v in comps.items():
+        lines.append(f"  - {k}: {v}")
+    if rule_breakdown.get("therapy_cliches_found"):
+        lines.append(f"  - Therapy clichés to remove: {rule_breakdown['therapy_cliches_found']}")
+    if rule_breakdown.get("ungrounded_citations"):
+        lines.append(
+            f"  - Citations that weren't in retrieved passages (likely hallucinated): "
+            f"{rule_breakdown['ungrounded_citations']}"
+        )
+    if not rule_breakdown.get("nondual_markers_found"):
+        lines.append("  - Response lacks explicit Advaita register; consider invoking "
+                     "concepts like sākṣī, adhyāsa, the two truths, etc.")
+    if not rule_breakdown.get("actionable_hits"):
+        lines.append("  - No concrete practice or this-week shift was offered.")
+    tier_counts = rule_breakdown.get("tier_counts", {})
+    if tier_counts:
+        lines.append(f"  - Selected passage tiers: {tier_counts} "
+                     f"(prefer primary + śaṅkara when both options exist).")
+
+    lines.append("")
+    lines.append("Rubric scores from Advaita-tradition examiner:")
+    for k, v in judge_rubric.items():
+        if isinstance(v, float):
+            lines.append(f"  - {k}: {v:.2f}")
+    lines.append("")
+    lines.append("Examiner critique:")
+    lines.append(critique.strip() or "(no critique returned)")
+    return "\n".join(lines)
+
+
+def gita_metric(
+    gold: dspy.Example,
+    pred: dspy.Prediction,
+    trace: Any = None,
+    pred_name: str | None = None,
+    pred_trace: Any = None,
+) -> dspy.Prediction:
+    """The GEPA-compatible metric.
+
+    Returns dspy.Prediction(score=..., feedback=...). The feedback string is
+    what GEPA's reflection LM ingests when rewriting prompts."""
+    user_q = getattr(gold, "user_question", "") if gold else ""
+
+    rule_score, rule_breakdown = rule_based_score(pred)
+    j_score, j_rubric, critique = judge_score(user_q, pred)
+
+    composite = RULE_WEIGHT * rule_score + JUDGE_WEIGHT * j_score
+    feedback = _format_feedback(rule_breakdown, j_rubric, critique)
+
+    return dspy.Prediction(score=composite, feedback=feedback)
+
+
+def quick_eval_score(
+    gold: dspy.Example,
+    pred: dspy.Prediction,
+    trace: Any = None,
+) -> float:
+    """A pure-float metric for `dspy.Evaluate` — same composite, no feedback."""
+    out = gita_metric(gold, pred, trace=trace)
+    return float(out.score)

optimize_gepa.py

@@ -0,0 +1,200 @@
+"""
+optimize_gepa.py — run GEPA reflective prompt evolution.
+
+GEPA (Genetic-Pareto) treats the program's prompts as an evolving population.
+At each step it:
+  1. Runs the current candidate(s) on a minibatch of training examples
+  2. Collects the (score, feedback) pairs from our metric
+  3. Asks a *reflection LM* to read the failures + feedback and propose a
+     mutated prompt
+  4. Evaluates the mutant; keeps it if it Pareto-dominates the parent on the
+     validation set
+  5. Repeats
+
+Because we wrote `gita_metric` to return rich textual feedback, the reflection
+LM has something substantive to chew on instead of just gradient signal.
+
+The dataset has no gold labels — that's deliberate. Our metric judges the
+prediction directly. This is the regime GEPA is designed for.
+
+Usage:
+    python optimize_gepa.py --auto medium
+    python optimize_gepa.py --max-metric-calls 300 --proxy-task-lm
+    python optimize_gepa.py --auto light --proxy-task-lm   # ~2-3 hrs vs 260 hrs
+
+Proxy task LM (--proxy-task-lm):
+    Runs GEPA with gpt-4o-mini as the task LM instead of Gemma 4. GEPA only
+    needs to evaluate prompt quality — it doesn't need the final inference model.
+    Optimized prompts are model-agnostic text and transfer back to Gemma 4 when
+    the saved program is loaded at inference time. ~20x speedup over Gemma thinking.
+"""
+
+from __future__ import annotations
+import argparse
+import json
+import random
+from pathlib import Path
+
+import dspy
+from dspy import GEPA
+
+import config
+from advisor import GitaAdvisor
+from dataset_generator import load_jsonl, to_dspy_examples
+import metrics as metrics_module
+from metrics import gita_metric, quick_eval_score
+
+
+def split(examples, val_frac: float, seed: int = 42):
+    rng = random.Random(seed)
+    shuffled = examples[:]
+    rng.shuffle(shuffled)
+    n_val = max(20, int(len(shuffled) * val_frac))
+    return shuffled[n_val:], shuffled[:n_val]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--dataset", default=str(config.DATASET_PATH))
+    ap.add_argument("--out", default=str(config.OPTIMIZED_PROGRAM_PATH))
+    ap.add_argument("--val-frac", type=float, default=0.2)
+    ap.add_argument(
+        "--auto",
+        choices=["light", "medium", "heavy"],
+        default="medium",
+        help="GEPA's auto-budget mode. 'light' for smoke-tests, 'medium' for "
+             "a real run, 'heavy' for an overnight run on a meaty box.",
+    )
+    ap.add_argument(
+        "--max-metric-calls",
+        type=int,
+        default=None,
+        help="Override --auto with an explicit metric-call budget.",
+    )
+    ap.add_argument("--track-stats", action="store_true", default=True)
+    ap.add_argument("--seed", type=int, default=42)
+    ap.add_argument(
+        "--proxy-task-lm",
+        action="store_true",
+        default=False,
+        help="Use gpt-4o-mini as the task LM during GEPA instead of Gemma 4. "
+             "~20x faster; optimized prompts transfer back to Gemma 4 at inference. "
+             "Requires OPENAI_API_KEY.",
+    )
+    args = ap.parse_args()
+
+    # Configure DSPy globally and grab the reflection LM
+    task_lm, reflection_lm = config.configure_dspy()
+
+    if args.proxy_task_lm:
+        # Override the task LM with gpt-4o-mini for the duration of this process.
+        # DSPy saves only prompt text (instructions + field descriptions), not the
+        # LM choice — so the optimized JSON loads cleanly onto Gemma 4 at inference.
+        task_lm = dspy.LM(model=config.PROXY_TASK_MODEL, **config.PROXY_TASK_LM_KWARGS)
+        dspy.configure(lm=task_lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False))
+        print(f"Task LM (proxy): {task_lm.model}  [GEPA optimization only]")
+    else:
+        print(f"Task LM:         {task_lm.model}")
+    print(f"Reflection LM:   {reflection_lm.model}")
+
+    # Use the reflection LM (gpt-4o) for judging instead of the task LM (Gemma).
+    # Gemma judging its own responses produces noisy, self-congratulatory scores;
+    # gpt-4o gives the reflection step the crisp, tradition-aware feedback it needs.
+    metrics_module.configure_judge(reflection_lm)
+    print(f"Judge LM:      {reflection_lm.model} (overriding task LM for judging)")
+
+    # Dataset
+    raw = load_jsonl(Path(args.dataset))
+    examples = to_dspy_examples(raw)
+    if len(examples) < 40:
+        print(f"[warn] Only {len(examples)} examples — generate more with "
+              f"`python dataset_generator.py --n 500`.")
+    train, val = split(examples, args.val_frac, seed=args.seed)
+    print(f"Train: {len(train)}   Val: {len(val)}")
+
+    # Student program
+    student = GitaAdvisor()
+
+    # More threads when hitting an API (no local GPU bottleneck).
+    num_threads = 16 if args.proxy_task_lm or config.TASK_LM_BACKEND == "gemini" else 4
+
+    # Optional: get a baseline number for context
+    print("\nEvaluating baseline (un-optimized) on validation set ...")
+    evaluator = dspy.Evaluate(
+        devset=val,
+        metric=quick_eval_score,
+        num_threads=num_threads,
+        display_progress=True,
+        display_table=0,
+    )
+    try:
+        baseline_result = evaluator(student)
+        baseline_score = float(baseline_result) if hasattr(baseline_result, "__float__") else baseline_result
+        print(f"Baseline score: {baseline_score}")
+    except Exception as e:
+        print(f"Baseline eval failed (continuing to optimization): {e}")
+
+    # GEPA
+    log_dir = str(config.ARTIFACTS_DIR / "gepa_logs")
+    gepa_kwargs = dict(
+        metric=gita_metric,
+        reflection_lm=reflection_lm,
+        track_stats=args.track_stats,
+        seed=args.seed,
+        # Show 6 training examples to the reflection LM per proposal step instead of
+        # the default 3 — our 12 domains need diversity to avoid domain-specific over-fit.
+        reflection_minibatch_size=6,
+        # API-backed runs (proxy or Gemini) can saturate many threads; local GPU is
+        # limited to 4 to avoid OOM / serialization on a single device.
+        num_threads=num_threads,
+        # When the task LM mangles a list field the reflection LM should know the format
+        # broke, not just see a low score with no explanation.
+        add_format_failure_as_feedback=True,
+        # Persist per-step scores and prompts for post-run inspection.
+        log_dir=log_dir,
+    )
+    if args.max_metric_calls is not None:
+        gepa_kwargs["max_metric_calls"] = args.max_metric_calls
+    else:
+        gepa_kwargs["auto"] = args.auto
+
+    print(f"\nStarting GEPA with {gepa_kwargs} ...")
+    optimizer = GEPA(**gepa_kwargs)
+
+    optimized = optimizer.compile(
+        student=student,
+        trainset=train,
+        valset=val,
+    )
+
+    # Save
+    out_path = Path(args.out)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    optimized.save(str(out_path))
+    print(f"\nSaved optimized program to {out_path}")
+
+    # Side-by-side eval
+    print("\nFinal eval on validation set ...")
+    final_result = evaluator(optimized)
+    final_score = float(final_result) if hasattr(final_result, "__float__") else final_result
+    print(f"Optimized score: {final_score}")
+
+    # Dump the optimized prompts for human inspection
+    inspect_path = out_path.with_suffix(".prompts.txt")
+    with inspect_path.open("w", encoding="utf-8") as f:
+        f.write("# Optimized prompts after GEPA\n\n")
+        for name, predictor in optimized.named_predictors():
+            sig = predictor.signature
+            f.write(f"## {name}\n")
+            f.write(f"### instructions\n{sig.instructions}\n\n")
+            f.write("### fields\n")
+            for fname, field in sig.fields.items():
+                desc = getattr(field.json_schema_extra, "get", lambda *_: "")("desc", "") \
+                    if hasattr(field, "json_schema_extra") else ""
+                f.write(f"- {fname}: {desc}\n")
+            f.write("\n---\n\n")
+    print(f"Wrote prompt inspection file to {inspect_path}")
+
+
+if __name__ == "__main__":
+    main()

parsers/gita_json.py

@@ -0,0 +1,236 @@
+"""
+parsers/gita_json.py — turn the gita/gita verse-indexed JSON into Verse records.
+
+The gita/gita repo (Unlicense, public-domain dedication) gives us four files
+on the static mirror:
+
+    chapters.json     — chapter metadata (number, name, summary)
+    verse.json        — per-verse Sanskrit + transliteration + word_meanings
+    translation.json  — per-verse English translations keyed by author_id
+    authors.json      — author metadata for the translations
+
+Why split parsing across multiple sources_registry entries
+----------------------------------------------------------
+We register `gita_json_core` (the verse text) and `gita_json_translations`
+(the English translations) as separate sources. Both happen to feed this one
+parser. The reason for the split is that translations come and go from the
+upstream repo whereas the core verse data is essentially fixed; isolating
+them lets us pin only what we need.
+
+Translator allowlist
+--------------------
+Not every translator in the gita/gita translations.json is public-domain.
+We hard-allowlist the ones we know are safe to redistribute. Anyone not on
+the list is silently skipped — adding more is a one-line change.
+"""
+
+from __future__ import annotations
+import json
+from pathlib import Path
+from typing import Iterable
+
+from corpus import Verse
+
+
+# ──────────────────────────── Translator allowlist ────────────────────────────
+# The keys are the author_id values used inside translation.json. The values
+# are display strings + the year we want to use for attribution.
+#
+# Why this list and not just "all translations":
+# - Some translators in the upstream repo (e.g. ISKCON Prabhupada) have
+#   active publisher rights that we shouldn't rely on regardless of how the
+#   upstream chose to license its compilation.
+# - Reducing translation count keeps the index lean. Three voices are plenty.
+#
+# If you want to add a translator, verify their public-domain status (death
+# year + 70 in most jurisdictions, or pre-1929 publication for US PD), then
+# add a row.
+ALLOWED_TRANSLATORS: dict[str, tuple[str, int | None]] = {
+    # Swami Sivananda — d. 1963 — works are widely shared by The Divine Life
+    # Society in keeping with their founder's non-commercial stance.
+    "sivananda":   ("Swami Sivananda", 1969),
+
+    # Swami Tejomayananda — modern; included only because some mirrors
+    # release these under permissive terms; double-check before relying on it.
+    # Disabled by default to be conservative.
+    # "tejomayananda": ("Swami Tejomayananda", 1995),
+
+    # Dr. S. Sankaranarayan — translation of Śaṅkara's Gītā Bhāṣya included
+    # in some forks of gita/gita; verify the specific edition. Off by default.
+    # "shankara":    ("Śaṅkara (tr. Sankaranarayan)", 1990),
+
+    # The verse text itself is not a "translation" per se but a copy of the
+    # critical text plus transliteration. We include it under the synthetic
+    # author key 'sanskrit'.
+    "sanskrit":    ("Sanskrit text + IAST", None),
+}
+
+
+# ──────────────────────────── Helpers ────────────────────────────
+def _verse_id(chapter: int, verse_no: int) -> str:
+    """Stable global key. Format: bhagavad_gita_<chap>_<verse>, zero-padded
+    to two digits so 1.10 sorts after 1.9 and lexical ordering matches numeric."""
+    return f"bhagavad_gita_{chapter:02d}_{verse_no:02d}"
+
+
+def _verse_ref(chapter: int, verse_no: int) -> str:
+    """Citation form used by the advisor in its replies."""
+    return f"BG {chapter}.{verse_no}"
+
+
+def _section_display(chapter_meta: dict) -> str:
+    name = chapter_meta.get("name_translation") or chapter_meta.get("name", "")
+    return f"Chapter {chapter_meta.get('chapter_number', '?')}: {name}"
+
+
+# ──────────────────────────── Parser entry point ────────────────────────────
+def parse(raw_dir_for_core: Path, raw_dir_for_translations: Path | None = None) -> Iterable[Verse]:
+    """Walk the gita/gita JSON files and yield Verse records.
+
+    Layout expected (after download_sources.py has run):
+        raw_dir_for_core/chapters.json
+        raw_dir_for_core/verse.json
+        [optionally]
+        raw_dir_for_translations/translation.json
+        raw_dir_for_translations/authors.json
+
+    If translations are not present, we still emit Verses with sanskrit +
+    transliteration + word_meanings; the `translation` field falls back to
+    the transliteration so the verse isn't content-empty. (Better: enable
+    the gita_json_translations source.)
+    """
+    chapters = _load(raw_dir_for_core / "chapters.json")
+    verses_raw = _load(raw_dir_for_core / "verse.json")
+
+    chapters_by_id = {c["chapter_number"]: c for c in chapters}
+
+    translations_by_verse: dict[int, dict[str, str]] = {}
+    authors_by_id: dict[str, str] = {}
+    if raw_dir_for_translations is not None:
+        translations_by_verse = _load_translations(raw_dir_for_translations / "translation.json")
+        authors_by_id = _load_authors(raw_dir_for_translations / "authors.json")
+
+    # Pick the best available translator from the allowlist, in priority order.
+    # First match wins. This keeps the index from carrying redundant English
+    # translations of the same verse.
+    translator_priority = ["sivananda", "sanskrit"]
+
+    for v in verses_raw:
+        chap_no = v["chapter_number"]
+        verse_no = v["verse_number"]
+        chap_meta = chapters_by_id.get(chap_no, {})
+        verse_id = _verse_id(chap_no, verse_no)
+
+        # Sanskrit text comes from the core file. The 'text' field has it
+        # in Devanāgarī, often with a trailing newline and verse number.
+        sanskrit = (v.get("text") or "").strip()
+        translit = (v.get("transliteration") or "").strip()
+        word_mean = (v.get("word_meanings") or "").strip()
+
+        # Try to attach an English translation
+        english = ""
+        translator_label = ""
+        v_translations = translations_by_verse.get(v.get("id") or v.get("externalId") or -1, {})
+        for key in translator_priority:
+            text = v_translations.get(key) or _translation_for(v_translations, key)
+            if text:
+                english = text.strip()
+                meta = ALLOWED_TRANSLATORS.get(key)
+                if meta:
+                    translator_label = meta[0]
+                break
+
+        # Fallback: if no English translation, use word-meanings as a substitute
+        # so the verse isn't content-empty. Better than nothing for retrieval,
+        # though enrichment will be poorer.
+        if not english:
+            english = word_mean or translit
+
+        yield Verse(
+            verse_id=verse_id,
+            work="bhagavad_gita",
+            work_display="Bhagavad Gītā",
+            verse_ref=_verse_ref(chap_no, verse_no),
+            tier="primary",
+            section=f"chapter_{chap_no:02d}",
+            section_display=_section_display(chap_meta),
+            translation=english,
+            translator=translator_label,
+            sanskrit=sanskrit,
+            transliteration=translit,
+            word_meanings=word_mean,
+            bhashya="",                  # Gītā Bhāṣya is brought in by the Sastry parser
+            bhashya_translator="",
+            source_key="gita_json_core",
+            license="unlicense",
+        )
+
+
+# ──────────────────────────── Internals ────────────────────────────
+def _load(path: Path):
+    with path.open(encoding="utf-8") as f:
+        return json.load(f)
+
+
+def _load_translations(path: Path) -> dict[int, dict[str, str]]:
+    """The translations file has one entry per (verse, author). Group them
+    by verse_id into a {verse_id: {author_id: text}} map.
+
+    Schema seen in the wild varies slightly between forks of gita/gita; we
+    cope by trying a few key names. If parsing fails entirely we return {}
+    and proceed without translations rather than blowing up the whole ingest.
+    """
+    if not path.exists():
+        return {}
+    try:
+        raw = _load(path)
+    except Exception as e:
+        print(f"[gita_json] failed to load translations: {e}")
+        return {}
+
+    out: dict[int, dict[str, str]] = {}
+    for row in raw:
+        vid = row.get("verse_id") or row.get("verseNumber") or row.get("verse_number_id") or row.get("id")
+        text = row.get("description") or row.get("text") or row.get("translation")
+        if vid is None or not text:
+            continue
+
+        # Skip non-English rows (Ramsukhdas Hindi etc.)
+        lang = (row.get("lang") or "").lower()
+        if lang and lang not in ("english", "en"):
+            continue
+
+        # Map the authorName (e.g. "Swami Sivananda") to an allowlist key
+        # ("sivananda") via case-insensitive substring matching. The numeric
+        # author_id field alone can't match the allowlist, which is why we
+        # prefer authorName here.
+        name_str = str(row.get("authorName") or row.get("author_id") or row.get("author") or "").strip()
+        matched_key = next(
+            (k for k in ALLOWED_TRANSLATORS if k.lower() in name_str.lower()),
+            None,
+        )
+        if matched_key is None:
+            continue
+        out.setdefault(int(vid), {})[matched_key] = text
+    return out
+
+
+def _load_authors(path: Path) -> dict[str, str]:
+    if not path.exists():
+        return {}
+    try:
+        raw = _load(path)
+    except Exception:
+        return {}
+    return {row.get("id"): row.get("name", "") for row in raw if row.get("id")}
+
+
+def _translation_for(v_translations: dict, author_key: str) -> str | None:
+    """Tolerant lookup: some files use 'sivananda', some 'Sivananda', etc."""
+    if author_key in v_translations:
+        return v_translations[author_key]
+    lk = author_key.lower()
+    for k, val in v_translations.items():
+        if str(k).lower() == lk:
+            return val
+    return None

parsers/sastry_archive.py

@@ -0,0 +1,249 @@
+"""
+parsers/sastry_archive.py — extract verse-attached Śaṅkara bhāṣya from
+Alladi Mahadeva Sastry's 1897 archive.org OCR text.
+
+What makes this harder than the gita_json parser
+-------------------------------------------------
+The gita/gita JSON gave us each verse already keyed by chapter and verse
+number. The Sastry archive.org file is OCR'd plain text — about 20 MB of
+running prose where the only structural cues are:
+
+    1. Chapter headings, formatted in caps like "SANKHYA YOGA." or
+       "CHAPTER II — SANKHYA YOGA"
+    2. Verse markers, which appear in two forms in the OCR:
+         - inline as "(II. 47.)" or "II. 47." after a translated verse
+         - as section headings like "47." or "Verse 47." preceding the bhāṣya
+    3. The rule that when a translated verse appears, Śaṅkara's commentary
+       follows immediately until the next verse marker.
+
+Add to that: OCR noise. "II" can become "11", "47" can become "4 7", periods
+become commas, glyphs get dropped. So the parser is forgiving — it tries
+several patterns and falls back gracefully.
+
+What we extract
+---------------
+For each verse we find, we yield a Verse with:
+    - tier='shankara'
+    - work='bhagavad_gita_bhashya'  (kept distinct from 'bhagavad_gita' so
+      the joiner in ingest_corpus.py knows to merge bhashya into the gita
+      verses by verse_ref)
+    - translation = the verse text as Sastry rendered it (handy as a second
+      English voice alongside Sivananda)
+    - bhashya = Śaṅkara's commentary, as Sastry translated it
+    - bhashya_translator = 'Alladi Mahadeva Sastry, 1897'
+
+Robustness strategy
+-------------------
+We don't try to be perfect. If a verse's bhāṣya is mis-attributed by ±1, the
+downstream enrichment step will produce paraphrases that don't quite fit, and
+we'll catch those during the spot-check pass on enriched output. The metric
+will also penalize ungrounded citations. The key invariant is: never silently
+emit a wrong (verse_id, bhashya) pair if we're uncertain — better to skip.
+"""
+
+from __future__ import annotations
+import re
+from dataclasses import replace
+from pathlib import Path
+from typing import Iterable
+
+from corpus import Verse
+
+
+# ──────────────────────────── Patterns ────────────────────────────
+# Roman numerals (allowing OCR substitutions: I↔1, V↔V, etc.)
+ROMAN = r"(?:[IVX1l]+|[ivx]+)"
+
+# A "verse marker" looks like "II. 47" or "(II. 47.)" or "47" alone in a section
+# heading. We try several shapes and let the most specific win.
+VERSE_INLINE = re.compile(
+    r"\(?\s*(?P<chap>" + ROMAN + r")\s*[\.\,]\s*(?P<verse>\d{1,3})\s*[\.\,]?\s*\)?",
+    re.IGNORECASE,
+)
+
+# Chapter heading: "CHAPTER II" or "II. SANKHYA YOGA" — uppercase-heavy lines
+CHAPTER_HEADING = re.compile(
+    r"^\s*(?:CHAPTER\s+)?(?P<roman>" + ROMAN + r")\.?\s+[A-Z][A-Z \-—]{4,}",
+    re.MULTILINE,
+)
+
+# Roman → arabic
+ROMAN_MAP = {
+    "I": 1, "II": 2, "III": 3, "IV": 4, "V": 5, "VI": 6, "VII": 7, "VIII": 8,
+    "IX": 9, "X": 10, "XI": 11, "XII": 12, "XIII": 13, "XIV": 14, "XV": 15,
+    "XVI": 16, "XVII": 17, "XVIII": 18,
+}
+
+
+def _to_arabic(token: str) -> int | None:
+    """Convert a possibly-noisy roman numeral to an int. OCR sometimes turns
+    'I' into '1' and 'II' into '11', so we accept both forms."""
+    t = token.upper().replace("L", "I").replace("0", "O")  # OCR substitutions
+    if t in ROMAN_MAP:
+        return ROMAN_MAP[t]
+    # Pure-arabic fallback (e.g. OCR rendered 'II' as '11')
+    if t.isdigit():
+        n = int(t)
+        if 1 <= n <= 18:
+            return n
+    return None
+
+
+# ──────────────────────────── Main parse ────────────────────────────
+def parse(raw_dir: Path) -> Iterable[Verse]:
+    """Walk Sastry archive.org text in raw_dir and yield Verse records.
+
+    Expected layout (after download_sources.py):
+        raw_dir/Bhagavad-Gita.with.the.Commentary.of.Sri.Shankaracharya_djvu.txt
+
+    The file is ~20 MB of OCR text. We stream it line-by-line, maintain the
+    current chapter as we encounter chapter headings, and at each verse marker
+    yield the accumulated text since the previous marker as the bhāṣya.
+    """
+    txts = list(raw_dir.glob("*_djvu.txt")) + list(raw_dir.glob("*.txt"))
+    if not txts:
+        print(f"[sastry] no .txt under {raw_dir}; did you download_sources.py?")
+        return
+
+    text = txts[0].read_text(encoding="utf-8", errors="replace")
+    text = _denoise(text)
+
+    # First pass: find every verse marker with its position and attempt to
+    # disambiguate the chapter from context. We collect (chap, verse, span)
+    # tuples in document order.
+    markers: list[tuple[int, int, int, int]] = []  # chap, verse, start, end
+    current_chapter = 1
+    last_pos = 0
+
+    # Walk chapter headings and verse markers together via merged iteration
+    events = []
+    for m in CHAPTER_HEADING.finditer(text):
+        c = _to_arabic(m.group("roman"))
+        if c is not None:
+            events.append(("chapter", m.start(), c))
+
+    for m in VERSE_INLINE.finditer(text):
+        c = _to_arabic(m.group("chap"))
+        try:
+            v = int(m.group("verse"))
+        except (ValueError, TypeError):
+            continue
+        if c is None or not (1 <= v <= 80):
+            continue
+        events.append(("verse", m.start(), c, v, m.end(), m.start("verse")))
+
+    events.sort(key=lambda e: e[1])
+
+    # Second pass: build (chapter, verse) → (start, end) spans, where each
+    # span is the bhāṣya from one marker to the next. We yield in document
+    # order with the chapter from the most recent chapter heading we saw.
+    last_marker_pos: int | None = None
+    last_chap: int | None = None
+    last_verse: int | None = None
+
+    for ev in events:
+        if ev[0] == "chapter":
+            current_chapter = ev[2]
+            continue
+        # ev: ("verse", start, chap, verse, end, verse_pos)
+        _, start, chap, verse, end, verse_pos = ev
+
+        # Only treat markers where the verse NUMBER appears near the start of
+        # its line — those are actual section headings. Inline cross-references
+        # like "(II. 47.)" mid-paragraph have the verse number well into the
+        # line and must not be treated as section boundaries.
+        verse_line_start = text.rfind("\n", 0, verse_pos) + 1
+        on_own_line = (verse_pos - verse_line_start) <= 8
+        if not on_own_line:
+            continue
+        current_chapter = chap
+
+        if last_marker_pos is not None and last_chap is not None and last_verse is not None:
+            bhashya_text = text[last_marker_pos:start].strip()
+            if bhashya_text:
+                yield _build_verse(
+                    chap=last_chap, verse=last_verse, body=bhashya_text,
+                )
+
+        last_marker_pos = end
+        last_chap = current_chapter
+        last_verse = verse
+
+    # Flush the trailing one
+    if last_marker_pos is not None and last_chap and last_verse:
+        tail = text[last_marker_pos:].strip()
+        if tail:
+            yield _build_verse(chap=last_chap, verse=last_verse, body=tail)
+
+
+# ──────────────────────────── Builders ────────────────────────────
+def _build_verse(chap: int, verse: int, body: str) -> Verse:
+    """The body lump contains both Sastry's English of the verse and Śaṅkara's
+    commentary, usually with the verse first (sometimes labeled) and the
+    commentary following. We make a *light* split heuristic: if the first
+    paragraph is short (≤ 400 chars) and ends near a period, treat it as the
+    verse translation; the rest is bhashya. If we can't split confidently,
+    we put everything into bhashya and leave translation empty — the gita_json
+    parser already gave us a translation by another translator."""
+    body = body.strip()
+    translation = ""
+    bhashya = body
+
+    # Heuristic split on the first blank-ish line within reasonable distance
+    para_break = re.search(r"\n\s*\n", body[:600])
+    if para_break and para_break.end() < 500:
+        head = body[:para_break.start()].strip()
+        tail = body[para_break.end():].strip()
+        # Accept the split only if the head looks like a verse: short-ish,
+        # not starting with a typical-bhashya opener like "This means" /
+        # "The meaning is" / "Here the Lord says".
+        if 30 < len(head) < 400 and not _looks_like_bhashya_opener(head):
+            translation, bhashya = head, tail
+
+    return Verse(
+        verse_id=f"bhagavad_gita_{chap:02d}_{verse:02d}",
+        work="bhagavad_gita_bhashya",
+        work_display="Bhagavad Gītā with Śaṅkara's Bhāṣya",
+        verse_ref=f"BG {chap}.{verse}",
+        tier="shankara",
+        section=f"chapter_{chap:02d}",
+        section_display=f"Chapter {chap}",
+        translation=translation,
+        translator="Alladi Mahadeva Sastry" if translation else "",
+        bhashya=bhashya,
+        bhashya_translator="Alladi Mahadeva Sastry, 1897",
+        source_key="sastry_gita_bhashya",
+        license="public_domain",
+    )
+
+
+def _looks_like_bhashya_opener(s: str) -> bool:
+    s = s.strip().lower()
+    openers = (
+        "this means", "the meaning is", "the sense is", "here the lord",
+        "here it is said", "the lord says", "the question may", "objection",
+        "the commentator",
+    )
+    return any(s.startswith(o) for o in openers)
+
+
+# ──────────────────────────── OCR de-noise ────────────────────────────
+def _denoise(text: str) -> str:
+    """Light cleanup. Aggressive normalization risks losing real signal —
+    we only fix patterns we're confident about."""
+    # Common OCR substitutions for Sanskrit diacritics losses won't matter
+    # for English-language retrieval; we leave Sanskrit fragments alone.
+
+    # Collapse runs of repeated punctuation that OCR hallucinated
+    text = re.sub(r"\.{3,}", ".", text)
+    text = re.sub(r" +\.", ".", text)
+
+    # Glue cross-line hyphens: "lib-\nerty" → "liberty"
+    text = re.sub(r"-\n([a-z])", r"\1", text)
+
+    # Normalize whitespace
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r"\n[ \t]+", "\n", text)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+
+    return text

requirements.txt

@@ -0,0 +1,11 @@
+dspy-ai>=2.6.0
+chromadb>=0.5.0
+sentence-transformers>=3.0.0
+openai>=1.40.0
+pydantic>=2.0
+tqdm>=4.66
+numpy>=1.26
+rich>=13.7
+unidecode>=1.3
+requests>=2.31
+gradio>=4.0

run_overnight.py

@@ -0,0 +1,230 @@
+"""
+run_overnight.py — orchestrates full GEPA optimization through light → medium,
+then saves prompts and runs a multi-question test suite.
+
+Usage:
+    python run_overnight.py [--skip-light] [--skip-medium]
+
+Writes a timestamped log to artifacts/overnight_run.log.
+"""
+from __future__ import annotations
+import argparse
+import subprocess
+import sys
+import time
+from datetime import datetime
+from pathlib import Path
+import json
+
+ROOT = Path(__file__).parent.resolve()
+LOG_PATH = ROOT / "artifacts" / "overnight_run.log"
+OPTIMIZED_PATH = ROOT / "artifacts" / "optimized_advisor.json"
+PROMPTS_PATH = ROOT / "artifacts" / "optimized_advisor.prompts.txt"
+RESULTS_PATH = ROOT / "artifacts" / "test_results.json"
+
+TEST_QUESTIONS = [
+    "I just got laid off and feel like nothing matters anymore.",
+    "I keep procrastinating on important work and feel guilty about it. How do I stop?",
+    "My relationship ended and I feel like I've lost my identity. Who am I without this person?",
+    "I'm terrified of death and can't stop thinking about it at night.",
+    "I have achieved everything I wanted — career, family, money — and still feel empty.",
+    "I feel angry at everyone around me but don't know why. How should I deal with this?",
+    "I can't stop comparing myself to others and feeling like I'm always falling short.",
+]
+
+
+def ts() -> str:
+    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+
+def log(msg: str, f=None):
+    line = f"[{ts()}] {msg}"
+    print(line, flush=True)
+    if f:
+        f.write(line + "\n")
+        f.flush()
+
+
+def run_phase(cmd: list[str], phase: str, logfile) -> bool:
+    log(f"=== STARTING {phase} ===", logfile)
+    log(f"Command: {' '.join(cmd)}", logfile)
+    start = time.time()
+    try:
+        proc = subprocess.Popen(
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            text=True,
+            cwd=str(ROOT),
+        )
+        for line in proc.stdout:
+            logfile.write(line)
+            logfile.flush()
+            # Echo key lines to terminal
+            if any(k in line for k in ["score", "GEPA", "Step", "ERROR", "Saved", "Train:", "Val:", "Baseline"]):
+                print(line, end="", flush=True)
+        proc.wait()
+        elapsed = time.time() - start
+        if proc.returncode == 0:
+            log(f"=== {phase} COMPLETED in {elapsed/60:.1f} min ===", logfile)
+            return True
+        else:
+            log(f"=== {phase} FAILED (exit {proc.returncode}) after {elapsed/60:.1f} min ===", logfile)
+            return False
+    except Exception as e:
+        log(f"=== {phase} ERROR: {e} ===", logfile)
+        return False
+
+
+def run_test_suite(logfile) -> dict:
+    log("=== STARTING TEST SUITE ===", logfile)
+    sys.path.insert(0, str(ROOT))
+
+    import config
+    from advisor import load_optimized
+    from metrics import gita_metric
+    import dspy
+    from concurrent.futures import ThreadPoolExecutor, as_completed
+
+    config.configure_dspy()
+
+    advisor = load_optimized()
+    n = len(TEST_QUESTIONS)
+
+    def run_one(i_q):
+        i, q = i_q
+        try:
+            pred = advisor(user_question=q, history=dspy.History(messages=[]))
+            gold = dspy.Example(user_question=q).with_inputs("user_question")
+            m = gita_metric(gold, pred)
+            return i, q, {
+                "question": q,
+                "score": round(float(m.score), 3),
+                "word_count": len(pred.response.split()),
+                "sources_cited": pred.sources_cited,
+                "response_excerpt": pred.response[:200],
+                "feedback_excerpt": m.feedback[:500],
+            }
+        except Exception as e:
+            return i, q, {"question": q, "error": str(e), "score": 0.0}
+
+    indexed = list(enumerate(TEST_QUESTIONS, 1))
+    results_map = {}
+    with ThreadPoolExecutor(max_workers=n) as pool:
+        futures = {pool.submit(run_one, iq): iq for iq in indexed}
+        for fut in as_completed(futures):
+            i, q, result = fut.result()
+            results_map[i] = result
+            if "error" in result:
+                log(f"  [{i}/{n}] ERROR: {result['error']}", logfile)
+            else:
+                log(f"  [{i}/{n}] score={result['score']:.3f}  wc={result['word_count']}  sources={result['sources_cited']}", logfile)
+
+    results = [results_map[i] for i in range(1, n + 1)]
+    avg = sum(r.get("score", 0) for r in results) / n
+    log(f"=== TEST SUITE DONE — avg score: {avg:.3f} ===", logfile)
+    return {"questions": results, "avg_score": round(avg, 3), "timestamp": ts()}
+
+
+def dump_prompts(logfile):
+    """Re-extract and log optimized prompts to a human-readable file."""
+    if not OPTIMIZED_PATH.exists():
+        log("  No optimized program found — skipping prompt dump.", logfile)
+        return
+
+    sys.path.insert(0, str(ROOT))
+    import config
+    from advisor import GitaAdvisor
+    config.configure_dspy()
+
+    advisor = GitaAdvisor()
+    try:
+        advisor.load(str(OPTIMIZED_PATH))
+    except Exception as e:
+        log(f"  Could not load optimized program: {e}", logfile)
+        return
+
+    lines = ["# Optimized Prompts after GEPA overnight run", f"# Extracted at {ts()}", ""]
+    for name, predictor in advisor.named_predictors():
+        sig = predictor.signature
+        lines.append(f"## {name}")
+        lines.append(f"### Instructions")
+        lines.append(sig.instructions or "(none)")
+        lines.append("")
+        lines.append("### Field descriptions")
+        for fname, field in sig.fields.items():
+            extras = field.json_schema_extra or {}
+            desc = extras.get("desc", "") if isinstance(extras, dict) else ""
+            lines.append(f"  {fname}: {desc}")
+        lines.append("")
+        lines.append("---")
+        lines.append("")
+
+    PROMPTS_PATH.write_text("\n".join(lines), encoding="utf-8")
+    log(f"  Prompts written to {PROMPTS_PATH}", logfile)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--skip-light", action="store_true")
+    ap.add_argument("--skip-medium", action="store_true")
+    ap.add_argument("--skip-tests", action="store_true")
+    args = ap.parse_args()
+
+    LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
+
+    with LOG_PATH.open("w", encoding="utf-8") as logfile:
+        log("=== OVERNIGHT GEPA RUN STARTED ===", logfile)
+        log(f"Dataset: {ROOT / 'data' / 'synthetic_questions.jsonl'}", logfile)
+        log(f"Output:  {OPTIMIZED_PATH}", logfile)
+
+        python = sys.executable
+
+        # ── Phase 1: Light ──
+        if not args.skip_light:
+            ok = run_phase(
+                [python, "optimize_gepa.py", "--auto", "light"],
+                "GEPA LIGHT",
+                logfile,
+            )
+            if not ok:
+                log("Light phase failed — stopping overnight run.", logfile)
+                sys.exit(1)
+            # Back up light result
+            if OPTIMIZED_PATH.exists():
+                import shutil
+                shutil.copy(OPTIMIZED_PATH, OPTIMIZED_PATH.with_suffix(".light.json"))
+                log(f"  Backed up light result to {OPTIMIZED_PATH.with_suffix('.light.json')}", logfile)
+        else:
+            log("Skipping light phase (--skip-light).", logfile)
+
+        # ── Phase 2: Medium ──
+        if not args.skip_medium:
+            ok = run_phase(
+                [python, "optimize_gepa.py", "--auto", "medium"],
+                "GEPA MEDIUM",
+                logfile,
+            )
+            if not ok:
+                log("Medium phase failed.", logfile)
+                # Don't exit — still dump whatever we have
+        else:
+            log("Skipping medium phase (--skip-medium).", logfile)
+
+        # ── Dump prompts ──
+        log("Extracting optimized prompts ...", logfile)
+        dump_prompts(logfile)
+
+        # ── Test suite ──
+        if not args.skip_tests:
+            test_results = run_test_suite(logfile)
+            RESULTS_PATH.write_text(json.dumps(test_results, indent=2, ensure_ascii=False), encoding="utf-8")
+            log(f"Test results written to {RESULTS_PATH}", logfile)
+        else:
+            log("Skipping test suite (--skip-tests).", logfile)
+
+        log("=== OVERNIGHT RUN COMPLETE ===", logfile)
+
+
+if __name__ == "__main__":
+    main()