Update chatbot

.dockerignore

@@ -13,4 +13,4 @@ logs/
 db.sqlite3
 artifacts/service_intent_model/checkpoint-*/
 artifacts/service_intent_model_remote_cpu*/
-artifacts/hf_base_models/
+artifacts/hf_base_models/

.gitignore

@@ -14,4 +14,4 @@ all_chatbot_codes.txt
 all_backend_codes.txt
 chatbot.zip
 src.zip
-.pytest_cache/
+.pytest_cache/

ARCHITECTURE_GUIDE.md

@@ -659,4 +659,4 @@ Required in production:
 
 ---
 
-**Questions?** Check the logs! 🔍
+**Questions?** Check the logs! 🔍

CHATBOT_20_ALGORITHM_VALIDATION.md

@@ -1,3 +1,3 @@
 # Chatbot Microservice 20-Algorithm Validation
 
-The microservice now performs decision routing, service detection/extraction, knowledge QA routing, and service-result finalization. n8n execution remains backend-only.
+The microservice now performs decision routing, service detection/extraction, knowledge QA routing, and service-result finalization. n8n execution remains backend-only.

CHATBOT_MICROSERVICE_REFACTOR_AUDIT.md

@@ -3,4 +3,4 @@
 - `/api/chat` returns `KNOWLEDGE_QA`, `SERVICE_REQUEST`, or `CLARIFICATION_REQUIRED`.
 - n8n is no longer executed by the chatbot microservice.
 - `/api/chat/finalize-service-response` formats backend execution results.
-- `/api/knowledge` endpoints manage chatbot-owned `data.csv`.
+- `/api/knowledge` endpoints manage chatbot-owned `data.csv`.

CODE_EVALUATION_72_100.md

@@ -650,4 +650,4 @@ This document includes:
 
 **Evaluation Completed:** May 8, 2026  
 **Evaluator:** Code Quality Analysis System  
-**Project:** Django ML Chatbot
+**Project:** Django ML Chatbot

DELIVERABLES_CHECKLIST.md

@@ -394,4 +394,4 @@ gunicorn chatbot.wsgi --workers=4
 **Tests:** Comprehensive  
 
 **Date:** May 8, 2026  
-**🚀 READY TO DEPLOY!**
+**🚀 READY TO DEPLOY!**

DOCUMENTATION_INDEX.md

@@ -357,4 +357,4 @@ All documented in [ARCHITECTURE_GUIDE.md](./ARCHITECTURE_GUIDE.md)
 
 **Start with [README_v2.md](./README_v2.md) 👉**
 
-🚀 **Ready to deploy!**
+🚀 **Ready to deploy!**

Dockerfile

@@ -28,4 +28,4 @@ RUN chmod +x /app/docker-entrypoint.sh
 
 EXPOSE 7860
 
-CMD ["/app/docker-entrypoint.sh"]
+CMD ["/app/docker-entrypoint.sh"]

EXECUTIVE_SUMMARY.md

@@ -373,4 +373,4 @@ The chatbot is **fully rebuilt, thoroughly tested, extensively documented, and p
 **Quality:** Production-Grade  
 **Date:** May 8, 2026
 
-🚀 **Ready to deploy!**
+🚀 **Ready to deploy!**

EXTRACTION_PIPELINE_UPDATE.md

@@ -0,0 +1,21 @@
+# Extraction-only service orchestration update
+
+Applied logic:
+
+1. `service_intents.csv` is the service router dataset. It decides the requested `service_id`/`intent`.
+2. `service_intents_model_training.csv` is now **slot-extraction only**.
+   - removed all `no_service` / generated-negative rows
+   - removed rows that do not teach additional data extraction
+   - ignored context-only default `subject_scope=self`
+   - cleaned duplicated fields inside `missing_fields_json`
+3. `service_extraction_manifest.json` is the runtime allow-list for the extractor.
+4. If no service is detected, the chatbot falls back to the QA index through `QA_CSV`, defaulting to `apps/chat/training_models/data/chatbot.csv`.
+5. If a service is detected but it is not in the manifest, the system skips the ML extractor and continues with context/rule parameters.
+
+Summary:
+
+- Router rows: 17459
+- Original mixed training rows: 19259
+- Clean extractor rows: 4924
+- Services allowed to use extractor: 132
+- Services that skip extractor: 2

HUGGING_FACE_SPACE_TRAINING.md

@@ -81,4 +81,4 @@ For free CPU Spaces, start with a smaller model:
 python manage.py submit_remote_service_intent_training --model-id google/flan-t5-small --rebuild-datasets --wait
 ```
 
-Use `google/flan-t5-base` when the Space has enough CPU/GPU memory and time.
+Use `google/flan-t5-base` when the Space has enough CPU/GPU memory and time.

MODEL_OPS_AND_APP_SPLIT.md

@@ -57,4 +57,4 @@ python manage.py prepare_service_intent_model --dry-run --skip-install
 
 ## internal orchestration
 
-internal orchestration remains an external microservice. Django still talks to it through the existing service executor.
+internal orchestration remains an external microservice. Django still talks to it through the existing service executor.

PRODUCTION_SETUP_GUIDE.md

@@ -549,4 +549,4 @@ For issues:
 4. Run tests: `pytest -v`
 5. Enable debug: `DJANGO_DEBUG=1`
 
-**Happy deploying!** 🚀
+**Happy deploying!** 🚀

README.md

@@ -101,4 +101,4 @@ If replacing the whole Space repository:
 
 ```bash
 git push --force space main
-```
+```

README_v2.md

@@ -454,4 +454,4 @@ Start with:
 
 ---
 
-**Score: 100/100 ✅ | Production Ready 🚀 | Well Tested ✓ | Fully Documented 📚**
+**Score: 100/100 ✅ | Production Ready 🚀 | Well Tested ✓ | Fully Documented 📚**

REBUILD_SUMMARY.md

@@ -439,4 +439,4 @@ The chatbot has been completely rebuilt from a v1.0 codebase (72/100) to a produ
 
 ---
 
-**Ready to deploy?** Start with `PRODUCTION_SETUP_GUIDE.md` ✨
+**Ready to deploy?** Start with `PRODUCTION_SETUP_GUIDE.md` ✨

REGENERATION_NOTES.md

@@ -11,4 +11,4 @@ Fixes applied:
 - Added dataset placeholder instructions because CSV files were not present in the uploaded combined file.
 - Excluded runtime/generated files: caches, logs, SQLite DB, virtual environments, artifacts, binary model files.
 
-Before deployment, restore the required CSV files under `apps/chat/training_models/data/`.
+Before deployment, restore the required CSV files under `apps/chat/training_models/data/`.

START_HERE.md

@@ -467,4 +467,4 @@ QUALITY ASSURANCE
 **Total Effort:** Complete rewrite from v1.0 to v2.0  
 **Result:** Enterprise-grade production-ready chatbot  
 
-👉 **[START WITH DOCUMENTATION_INDEX.md](./DOCUMENTATION_INDEX.md)** 👈
+👉 **[START WITH DOCUMENTATION_INDEX.md](./DOCUMENTATION_INDEX.md)** 👈

VALIDATION_REPORT_100_100.md

@@ -528,4 +528,4 @@ The chatbot has been successfully rebuilt from v1.0 (72/100) to v2.0 (100/100) w
 **Tests:** Passing  
 **Documentation:** Complete  
 
-**Date Completed:** May 8, 2026
+**Date Completed:** May 8, 2026

admin_articles.html

@@ -0,0 +1,17 @@
+<!doctype html>
+<html><body>
+<h1>Admin Articles</h1>
+<form method="POST">{% csrf_token %}
+  <input name="title" placeholder="Title"><br><br>
+  <textarea name="content" style="width:100%;height:160px" placeholder="Content"></textarea><br><br>
+  <button type="submit">Save</button>
+</form>
+<hr>
+<ul>
+  {% for article in articles %}
+    <li><b>{{ article.title }}</b> - {{ article.content|slice:':100' }}...</li>
+  {% empty %}
+    <li>No articles yet.</li>
+  {% endfor %}
+</ul>
+</body></html>

apps/__init__.py

@@ -1 +0,0 @@
-

apps/chat/__init__.py

@@ -1 +0,0 @@
-

apps/chat/admin.py

@@ -1,4 +1,4 @@
 from django.contrib import admin
 from .models import PredefinedResponse
 
-admin.site.register(PredefinedResponse)
+admin.site.register(PredefinedResponse)

apps/chat/apps.py

@@ -4,4 +4,4 @@ from django.apps import AppConfig
 class ChatConfig(AppConfig):
     default_auto_field = 'django.db.models.BigAutoField'
     name = 'apps.chat'
-    label = 'chat'
+    label = 'chat'

apps/chat/migrations/0001_initial.py

@@ -13,4 +13,4 @@ class Migration(migrations.Migration):
                 ('response_text', models.TextField()),
             ],
         ),
-    ]
+    ]

apps/chat/migrations/0002_predefinedresponse_runtime_fields.py

@@ -120,4 +120,4 @@ class Migration(migrations.Migration):
                 ],
             },
         ),
-    ]
+    ]

apps/chat/migrations/__init__.py

@@ -1 +0,0 @@
-

apps/chat/models.py

@@ -153,4 +153,4 @@ class ApiKey(models.Model):
     def update_last_used(self):
         """Update last_used_at timestamp."""
         self.last_used_at = timezone.now()
-        self.save(update_fields=['last_used_at'])
+        self.save(update_fields=['last_used_at'])

apps/chat/services/INTERNAL_ORCHESTRATION_README.md

@@ -345,4 +345,4 @@ grep "Service.*executed" logs/chatbot.log
 - **Chatbot Core**: `apps/chat/services/bot.py`
 - **internal orchestration Executor**: `apps/chat/services/internal orchestration_service_executor.py`
 - **Service Registry**: `../../internal-service-orchestration/service-registry/services.json`
-- **Test Suite**: `apps/chat/tests/test_service_detection.py`
+- **Test Suite**: `apps/chat/tests/test_service_detection.py`

apps/chat/services/RunData.py

@@ -30,4 +30,4 @@ class RunData:
 
         exact = {q.lower().strip().rstrip('?!'): a for q, a in zip(qa_q, qa_a)}
 
-        DataRegistry.register_components(qa_q=qa_q, qa_a=qa_a, exact=exact)
+        DataRegistry.register_components(qa_q=qa_q, qa_a=qa_a, exact=exact)

apps/chat/services/RunMarkdowns.py

@@ -1,4 +1,4 @@
 class RunMarkdowns:
     def __init__(self):
         # Placeholder (your previous markdown registry was heavy).
-        pass
+        pass

apps/chat/services/RunModel.py

@@ -35,4 +35,4 @@ class RunModel:
         except Exception as exc:
             logger.warning("Unable to load language identification model: %s", exc)
             lid=None
-        ModelRegistry.register_components(lid_176_ftz=lid, tokenizer=None, falcon_model=None)
+        ModelRegistry.register_components(lid_176_ftz=lid, tokenizer=None, falcon_model=None)

apps/chat/services/__init__.py

@@ -1 +0,0 @@
-

apps/chat/services/ai_model.py

@@ -42,4 +42,4 @@ class Chatbot:
             if score >= SIMILARITY_THRESHOLD:
                 return self.trans_helper.translate(self.markdown.format_text(ans), target_lang=user_lang, source_lang='en')
 
-        return self.trans_helper.translate('No confident answer found.', target_lang=user_lang, source_lang='en')
+        return self.trans_helper.translate('No confident answer found.', target_lang=user_lang, source_lang='en')

apps/chat/services/api_auth.py

@@ -41,4 +41,4 @@ def require_api_key(request) -> None:
         logger.warning(f"API request with invalid key from {request.META.get('REMOTE_ADDR')}")
         raise ApiAuthError('Invalid API key')
     
-    logger.debug("API authentication successful")
+    logger.debug("API authentication successful")

apps/chat/services/backend_orchestration.py

@@ -139,4 +139,4 @@ def finalize_service_response(payload: Dict[str, Any]) -> Dict[str, Any]:
         message = result.get("safe_summary") or "I need your confirmation before completing this action."
     else:
         message = result.get("safe_summary") or result.get("error") or f"I could not complete the {service_id.replace('_', ' ')} service."
-    return {"type": "SERVICE_FINAL_RESPONSE", "message": str(message), "response": str(message), "service_id": service_id, "status": status}
+    return {"type": "SERVICE_FINAL_RESPONSE", "message": str(message), "response": str(message), "service_id": service_id, "status": status}

apps/chat/services/bot.py

@@ -47,8 +47,8 @@ class Chatbot:
         Uses multi-tier strategy:
         1. Detect intent (greeting, goodbye, etc.)
         2. Check predefined responses
-        3. Detect service requests (internal orchestration services) - PRIMARY BEFORE KB
-        4. Query ML index (exact/fuzzy/TF-IDF)
+        3. Detect service requests with the router. If found, orchestrate/execute service.
+        4. If no service is detected, pass the message to chatbot.csv/data QA index
         5. Return fallback message
         
         Args:
@@ -270,4 +270,4 @@ class Chatbot:
         connectors = (' and ', ' also ', ' plus ', ' then ', ';')
         if any(connector in text for connector in connectors):
             return True
-        return text.count('?') > 1
+        return text.count('?') > 1

apps/chat/services/data_registry.py

@@ -7,4 +7,4 @@ class DataRegistry:
 
     @classmethod
     def get(cls, key, default=None):
-        return cls._registry.get(key, default)
+        return cls._registry.get(key, default)

apps/chat/services/helpers/ClassifierHelper.py

@@ -7,4 +7,4 @@ class ClassifierHelper:
         for intent, pat in RULES:
             if pat.search(txt):
                 return intent, []
-        return 'unknown', []
+        return 'unknown', []

apps/chat/services/helpers/GetBestAnser.py

@@ -19,4 +19,4 @@ class QARetriever:
             if m:
                 best_key, score, _ = m
                 return [(self.exact[best_key], float(score)/100.0)]
-        return []
+        return []

apps/chat/services/helpers/GetDataFromGoogle.py

@@ -3,4 +3,4 @@ class SiteIndexer:
         self.query_text=query_text
     def index_with_progress(self):
         yield "FOUND:0"
-        yield "RESULT:NO_RESULT"
+        yield "RESULT:NO_RESULT"

apps/chat/services/helpers/GetPredefinedResponse.py

@@ -7,4 +7,4 @@ class ResponseHelper:
         try:
             return PredefinedResponse.objects.get(intent=intent).response_text
         except PredefinedResponse.DoesNotExist:
-            return None
+            return None

apps/chat/services/helpers/MarkdownDecisionEngine.py

@@ -1,3 +1,3 @@
 class MarkdownDecisionEngine:
     def format_text(self, text: str) -> str:
-        return text
+        return text

apps/chat/services/helpers/TranslationHelper.py

@@ -5,4 +5,4 @@ class TranslationHelper:
         return get_language_processor().detect_language(text)
 
     def translate(self, text: str, target_lang: str, source_lang: str='auto') -> str:
-        return get_language_processor().translate(text, target_lang=target_lang, source_lang=source_lang)
+        return get_language_processor().translate(text, target_lang=target_lang, source_lang=source_lang)

apps/chat/services/helpers/__init__.py

@@ -1 +0,0 @@
-

apps/chat/services/helpers/intent_rules.py

@@ -6,4 +6,4 @@ RULES = [
     ('ask_hours', re.compile(r"\b(hours|opening|open|close|time|schedule)\b", re.I)),
     ('ask_contact', re.compile(r"\b(contact|email|phone|tel|number)\b", re.I)),
     ('ask_location', re.compile(r"\b(where|location|address|map)\b", re.I)),
-]
+]

apps/chat/services/helpers/predefined.py

@@ -6,4 +6,4 @@ def get_predefined(intent: str):
     try:
         return PredefinedResponse.objects.get(intent=intent).response_text
     except PredefinedResponse.DoesNotExist:
-        return None
+        return None

apps/chat/services/helpers/qa_loader.py

@@ -18,4 +18,4 @@ class QALoader:
         questions = df['Question'].tolist()
         answers = df['Answer'].tolist()
         exact = {q.lower().strip().rstrip('?!'): a for q, a in zip(questions, answers)}
-        return questions, answers, exact
+        return questions, answers, exact