Upload folder using huggingface_hub

LICENSE

@@ -0,0 +1,262 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of control, an entity
+      is "controlled by" another entity if it has the power, directly or
+      indirectly, to cause the direction or management of such entity,
+      whether by contract or otherwise, or ownership of fifty percent (50%)
+      or more of the outstanding shares, or beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (which shall not include combinations of Works unless specifically
+      authorized by the License).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based upon (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of the License, a Derivative Work shall include any work that is based
+      upon a copy of the existing Work.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control
+      systems, and issue tracking systems that are managed by, or on behalf
+      of, the Licensor for the purpose of discussing and improving the Work,
+      but excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to use, reproduce, modify, create derivative works,
+      distribute, sublicense, and/or sell copies of the Work, and to
+      permit persons to whom the Work is furnished to do so, subject to
+      the following conditions:
+
+      The above copyright notice and this permission notice shall be
+      included in all copies or substantial portions of the Work.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright notices to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Support. You may choose to offer, and to
+      charge a fee for, warranty, support, indemnity or other liability
+      obligations and/or rights consistent with this License. However, in
+      accepting such obligations, You may act only on Your own behalf and on
+      Your sole responsibility, not on behalf of any other Contributor, and
+      only if You agree to indemnify, defend, and hold each Contributor
+      harmless for any liability incurred by, or claims asserted against,
+      such Contributor by reason of your accepting any such warranty or support.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same page as the copyright notice for easier identification within
+      third-party archives.
+
+   Copyright 2025 CUGA
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+---
+
+# NOTICE
+
+This project includes code from multiple open source projects:
+
+## BrowserGym
+Copyright 2024 ServiceNow
+Licensed under Apache License 2.0
+Source: https://github.com/ServiceNow/BrowserGym
+
+Portions of this project are derived from BrowserGym, including:
+- cuga/backend/browser_env/browser/chat_async.py
+- cuga/backend/browser_env/page_understanding/tranformer_utils/transform_utils.py
+- cuga/backend/browser_env/page_understanding/tranformer_utils/dom_transform_utils.py
+- cuga/backend/browser_env/browser/gym_env_async.py
+- cuga/backend/browser_env/browser/gym_obs/obs.py
+- cuga/backend/browser_env/browser/gym_obs/obs_async.py
+- cuga/backend/browser_env/browser/gym_obs/extract_chrome_extension.py
+- cuga/backend/browser_env/browser/env.py
+- cuga/backend/browser_env/browser/extension_env_async.py
+- cuga/backend/browser_env/browser/open_ended_async.py
+- cuga/backend/browser_env/browser/gym_env.py
+- cuga/backend/browser_env/browser/gym_obs/javascript/frame_unmark_elements.js
+- cuga/backend/browser_env/browser/gym_obs/javascript/frame_mark_elements.js
+- cuga/backend/browser_env/browser/utils_async.py
+
+## browser-use  
+Copyright (c) 2024 Gregor Zunic
+Licensed under MIT License
+Source: https://github.com/browser-use/browser-use
+
+Portions of this project are derived from browser-use, including:
+- frontend-workspaces/extension/src/content/page_analysis/CachedXPathBuilder.ts
+- frontend-workspaces/extension/src/content/page_analysis/constants.ts
+- frontend-workspaces/extension/src/content/page_analysis/dom_tree_module.ts
+- frontend-workspaces/extension/src/content/page_analysis/DomCache.ts
+- frontend-workspaces/extension/src/content/page_analysis/DomTree.ts
+- frontend-workspaces/extension/src/content/page_analysis/ElementHighlighter.ts
+- frontend-workspaces/extension/src/content/page_analysis/NodeElementCollector.ts
+- frontend-workspaces/extension/src/content/page_analysis/NodeHelper.ts
+- frontend-workspaces/extension/src/content/page_analysis/PageHighlighter.ts
+- frontend-workspaces/extension/src/content/page_analysis/types.d.ts
+
+
+## LangChain
+Copyright (c) 2025 LangChain
+Licensed under MIT License
+Source: https://github.com/langchain-ai/langgraph
+
+
+Portions of this project derived from LangChain include:
+- cuga/backend/cuga_graph/nodes/api/code_agent/code_act_agent.py
+
+## Original Work
+Copyright 2025 CUGA
+Licensed under Apache License 2.0
+
+All original code and modifications are licensed under Apache License 2.0.

policies/POLICIES.md

@@ -0,0 +1,163 @@
+# CUGA Policies for BPO Benchmark
+
+This document tracks all policies and their measured impact on evaluation scores.
+
+## Measured Results (5 runs each, clean Milvus DB)
+
+### Per-run scores
+
+```text
+Config          Run1 Run2 Run3 Run4 Run5   Mean   pass@5  pass^5
+──────────────────────────────────────────────────────────────────
+No policies       12   13   14   13   12   12.8    15/26   10/26
+5 policies        21   23   22   21   20   21.4    23/26   19/26
+```
+
+### Aggregate metrics
+
+| Metric | No Policies | 5 Policies | Delta |
+|---|---|---|---|
+| Mean score | 12.8/26 (49.2%) | 21.4/26 (82.3%) | +33.1pp |
+| pass@5 | 15/26 (57.7%) | 23/26 (88.5%) | +30.8pp |
+| pass^5 | 10/26 (38.5%) | 19/26 (73.1%) | +34.6pp |
+
+### Per-task breakdown
+
+```text
+Task     NoPol  @5 ^5  |  Pol   @5 ^5  Delta  Category
+────     ─────  ── ──  |  ───   ── ──  ─────  ────────
+   1      5/5   Y  Y   |  4/5   Y  -     -1   regression (flaky)
+   2      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+   3      1/5   Y  -   |  0/5   -  -     -1   regression
+   4      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+   5      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+   6      0/5   -  -   |  3/5   Y  -     +3   IMPROVED (flaky)
+   7      0/5   -  -   |  0/5   -  -      =   stable fail
+   8      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+   9      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #3)
+  10      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+  11      4/5   Y  -   |  5/5   Y  Y     +1   IMPROVED (stabilized)
+  12      3/5   Y  -   |  5/5   Y  Y     +2   IMPROVED (Policy #2)
+  13      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+  14      4/5   Y  -   |  5/5   Y  Y     +1   IMPROVED (Policy #2)
+  15      0/5   -  -   |  1/5   Y  -     +1   IMPROVED (flaky)
+  16      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #1)
+  17      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+  18      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+  19      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #1)
+  20      5/5   Y  Y   |  5/5   Y  Y      =   stable pass
+  21      0/5   -  -   |  4/5   Y  -     +4   IMPROVED (Policy #1, flaky)
+  22      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #1)
+  23      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #1)
+  24      2/5   Y  -   |  5/5   Y  Y     +3   IMPROVED (Policy #1/#2)
+  25      0/5   -  -   |  0/5   -  -      =   stable fail
+  26      0/5   -  -   |  5/5   Y  Y     +5   IMPROVED (Policy #4)
+────     ─────  ── ──  |  ───   ── ──
+TOTALS         15  10  |       23  19
+```
+
+---
+
+## Policy #1: API Capability Boundaries [IMPLEMENTED]
+
+**Type:** Playbook
+**File:** `api_capability_boundaries.md`
+**Triggers:** Keywords + natural language (threshold 0.65, priority 90)
+
+Teaches the agent to recognize when the available APIs cannot answer a question.
+Lists what the APIs can and cannot do, and instructs the agent to decline
+out-of-scope requests directly instead of asking for a requisition ID or calling
+irrelevant tools.
+
+**Failure pattern addressed:** The agent would ask for a requisition ID or call
+random APIs for queries about job descriptions, time-to-fill, geography
+filtering, SLA deadlines, funnel timing, and job-card details — none of which
+are supported by any API.
+
+**Tasks fixed:** 16, 19, 21, 22, 23 (all 0/5 → 4-5/5)
+
+---
+
+## Policy #2: Error-Prone Tool Warnings [IMPLEMENTED]
+
+**Type:** Tool Guide
+**File:** `error_tool_warnings.md`
+**Target:** 19 error-prone tools (prepended to their descriptions)
+
+Prepends a warning to the descriptions of the 19 known-unreliable tools (those
+that return 503s, schema violations, type mismatches). Steers the agent toward
+the 13 reliable core tools and teaches it to recover gracefully when an error
+tool is called.
+
+**Failure pattern addressed:** The agent would call tools like `funnel_status`
+(503 error), `model_registry` (wrong data), or `source_recommendation_summary`
+(incomplete shortcut) instead of using the correct granular APIs.
+
+**Tasks fixed:** 12 (3/5 → 5/5), 14 (4/5 → 5/5), 24 (2/5 → 5/5)
+
+---
+
+## Policy #3: Multi-API Reasoning [IMPLEMENTED]
+
+**Type:** Playbook
+**File:** `multi_api_reasoning.md`
+**Triggers:** Keywords + natural language (threshold 0.65, priority 80)
+
+Instructs the agent on when to call multiple specific APIs instead of relying on
+a single summary endpoint. Provides a mapping from question type to the correct
+specific tool, and clarifies the difference between "total requisitions used for
+computation" (`definitions-and-methodology`) vs "similar requisitions analysed"
+(`metadata-and-timeframe`).
+
+**Failure pattern addressed:** The agent would use the summary shortcut tool for
+multi-metric questions, or confuse which API returns the requisition count.
+
+**Tasks fixed:** 9 (0/5 → 5/5 — now correctly returns 1047 from
+`definitions-and-methodology` instead of 40 from `metadata-and-timeframe`)
+
+---
+
+## Policy #4: Average vs Total Calculations [IMPLEMENTED]
+
+**Type:** Playbook
+**File:** `average_vs_total.md`
+**Triggers:** Keywords + natural language (threshold 0.65, priority 70)
+
+Teaches the agent that when the user asks for "average" or "typical" values, it
+must compute a per-requisition average by dividing the total by the count of
+similar requisitions, rather than returning the raw total.
+
+**Failure pattern addressed:** The agent would return the total candidate count
+(2913) when asked "how many candidates do we usually get" instead of computing
+the average (2913 / 40 = ~73).
+
+**Tasks fixed:** 26 (0/5 → 5/5)
+
+---
+
+## Policy #5: Missing Requisition ID vs Unsupported Query [IMPLEMENTED]
+
+**Type:** Playbook
+**File:** `missing_req_id_vs_unsupported.md`
+**Triggers:** Keywords + natural language (threshold 0.60, priority 85)
+
+Helps the agent distinguish between "I need a requisition ID to answer this"
+(answerable but missing context) vs "This can't be answered regardless of
+requisition ID" (unsupported by any API). Reinforces Policy #1 for edge cases.
+
+**Failure pattern addressed:** The agent would ask for a requisition ID even
+when the question was about something no API supports.
+
+**Tasks fixed:** Overlaps with Policy #1; provides reinforcement for edge cases.
+
+---
+
+## Remaining Failing Tasks
+
+| Task | Pass rate (no pol) | Pass rate (5 pol) | Issue | Notes |
+|---|---|---|---|---|
+| 1 | 5/5 | 4/5 | Flaky regression | May be LLM non-determinism |
+| 3 | 1/5 | 0/5 | Agent provides incomplete source data | Regression — policies may over-constrain |
+| 7 | 0/5 | 0/5 | LLM judge scores correct behavior as 0 | Judge issue, not agent issue |
+| 15 | 0/5 | 1/5 | Calls too few APIs, misidentifies negative-SLA skills | Complex multi-part question |
+| 25 | 0/5 | 0/5 | Calls error tool for invalid requisition ID instead of declining | Edge case not caught by Policy #1 triggers |

policies/api_capability_boundaries.md

@@ -0,0 +1,47 @@
+# API Capability Boundaries
+
+Before answering any question, verify that the available APIs can actually provide the needed data.
+If they cannot, tell the user directly — do NOT attempt to cobble together an answer from unrelated endpoints, and do NOT ask for a requisition ID when the query is fundamentally unsupported.
+
+## What the APIs CAN do
+
+The available tool suite covers two domains:
+
+### Candidate Source Analytics
+- SLA percentage per sourcing channel
+- Total hires per sourcing channel
+- Candidate volume and share per sourcing channel
+- Funnel conversion rates (review %, interview %, offer acceptance %) per source
+- Composite source recommendation summary
+- Metadata: data timeframe, last update date, number of similar requisitions analysed
+- Definitions and methodology: metric definitions, total requisition count used for computation, ML models involved
+
+### Skills Analytics
+- Skill-level statistical analysis (historical counts, SLA correlation)
+- Skill impact on fill rate
+- Skill impact on SLA (delta with/without the skill)
+- Skill relevance justification
+- Data sources and ML models used for recommendations
+- Successful posting criteria and benchmarks
+
+## What the APIs CANNOT do
+
+The following capabilities are NOT available through any API. If the user asks for any of these, explain that the current API suite does not support it:
+
+- **Job description text**: No API returns or accepts raw job description content. You cannot read, optimise, or rewrite a job description.
+- **Time-to-fill metrics**: No API provides time-to-fill data, whether overall or broken down by source.
+- **Geographic or channel filtering**: No API supports filtering by country, region, or posting channel (internal vs external).
+- **Live requisition status or SLA countdowns**: The APIs provide historical/aggregate analytics, not real-time status tracking or deadline monitoring.
+- **Stage-by-stage funnel timing**: No API returns average days spent in each pipeline stage or candidate counts per stage over time.
+- **Full job-card details**: No API returns comprehensive requisition details like title, location, hiring-manager name, or contact information. The APIs focus on aggregate analytics, not individual job metadata.
+- **Cross-requisition listing or search**: The APIs analyse one requisition at a time against historical data. They cannot list, search, or filter across all open requisitions.
+
+## How to respond when a query is out of scope
+
+When you determine that a question cannot be answered with the available APIs:
+
+1. State clearly that the current APIs do not provide the requested data
+2. Be specific about what is missing (e.g., "the APIs don't expose time-to-fill broken down by source")
+3. Do NOT ask for a requisition ID — providing one would not help
+4. Do NOT call any API tools — the answer is that the data is unavailable
+5. Do NOT fabricate or infer data that no API returned

policies/average_vs_total.md

@@ -0,0 +1,20 @@
+# Average vs Total Calculations
+
+When the user asks for "average", "typical", "usually", or "per posting" values, you must compute an average — do not return a raw total.
+
+## How to compute averages
+
+1. Get the total metric value from the relevant API (e.g., total candidate volume from `candidate_source_candidate_volume_by_source`)
+2. Get the number of similar requisitions from `candidate_source_metadata_and_timeframe`
+3. Divide the total by the number of similar requisitions to get the per-requisition average
+4. Report the average, not the total
+
+## Example
+
+If the user asks "How many candidates do we usually get for postings similar to X?":
+- Total candidates across all sources = 2913
+- Number of similar requisitions = 40
+- Average = 2913 / 40 = ~73 candidates per posting
+- Report: "On average, similar postings attract 73 candidates"
+
+Do NOT report 2913 as the answer — that is the total, not the average.

policies/error_tool_warnings.md

@@ -0,0 +1,15 @@
+# Error-Prone Tool Warning
+
+WARNING: This tool is known to be unreliable. It may return HTTP errors (e.g. 503 Service Unavailable), schema violations, type mismatches, or unexpected data formats.
+
+Before using this tool, check whether one of the 13 core reliable tools can answer the question instead:
+
+**Reliable Candidate Source tools:** candidate_source_sla_per_source, candidate_source_total_hires_by_source, candidate_source_candidate_volume_by_source, candidate_source_funnel_conversion_by_source, candidate_source_metadata_and_timeframe, candidate_source_definitions_and_methodology, candidate_source_source_recommendation_summary
+
+**Reliable Skills tools:** skills_skill_analysis, skills_skill_impact_fill_rate, skills_skill_impact_sla, skills_skill_relevance_justification, skills_successful_posting_criteria, skills_data_sources_used
+
+If this tool returns an error or unexpected data:
+- Do NOT report the raw error message to the user
+- Do NOT retry the same tool
+- Check if a reliable tool can provide the needed data
+- If no reliable tool can help, tell the user the data is not available through the current APIs

policies/missing_req_id_vs_unsupported.md

@@ -0,0 +1,28 @@
+# Missing Requisition ID vs Unsupported Query
+
+When a question does not include a requisition ID, determine whether providing one would actually help before asking for it.
+
+## Ask for a requisition ID when:
+
+The question is about something the APIs support but needs a specific requisition to look up:
+- SLA performance by source
+- Candidate volume or hires by source
+- Skill analysis or skill impact
+- Funnel conversion rates
+- Data sources or methodology used
+- Metadata and timeframe
+
+These all require a requisition ID to return useful results.
+
+## Do NOT ask for a requisition ID when:
+
+The question is about something no API supports regardless of requisition ID:
+- Job description text (reading, optimizing, rewriting)
+- Time-to-fill metrics (overall or by source)
+- Geographic or location-based filtering
+- Live requisition status or SLA deadline countdowns
+- Stage-by-stage funnel timing (days in each stage)
+- Full job-card details (title, location, hiring-manager info)
+- Listing or searching across all open requisitions
+
+For these, explain directly that the current APIs do not support the request. Asking for a requisition ID would be misleading because providing one would not help.

policies/multi_api_reasoning.md

@@ -0,0 +1,34 @@
+# Multi-API Reasoning
+
+When a question asks about multiple dimensions of performance, call the individual specific APIs rather than relying on a single summary endpoint. Summary tools give a quick overview but often lack the granular detail needed for a complete answer.
+
+## Source Performance Questions
+
+When comparing or recommending sources across multiple metrics (SLA, volume, conversion, hires), call the specific tools:
+
+- **SLA performance** → `candidate_source_sla_per_source`
+- **Candidate volume and share** → `candidate_source_candidate_volume_by_source`
+- **Funnel conversion rates** (review %, interview %, offer acceptance %) → `candidate_source_funnel_conversion_by_source`
+- **Total hires by source** → `candidate_source_total_hires_by_source`
+
+Do NOT rely solely on `candidate_source_source_recommendation_summary` when the question asks for specific metrics like SLA percentages, offer acceptance rates, or conversion rates. The summary tool is useful for a quick recommendation but does not contain all granular metrics.
+
+## Requisition Count and Sample Size Questions
+
+There are two different "counts" in the system — do not confuse them:
+
+- **"How many requisitions were used to compute these metrics"** or **"sample size"** → use `candidate_source_definitions_and_methodology`, which returns the total number of requisitions used for computation across the system
+- **"How many similar requisitions were analysed"** → use `candidate_source_metadata_and_timeframe`, which returns the count of requisitions similar to the given one
+
+These return different numbers. Read the question carefully to determine which one is being asked for.
+
+## Skill Analysis Questions
+
+When a question asks about skill impact across multiple dimensions (SLA, fill rate, relevance):
+
+- **SLA impact** → `skills_skill_impact_sla`
+- **Fill rate impact** → `skills_skill_impact_fill_rate`
+- **Historical effectiveness and statistical analysis** → `skills_skill_analysis`
+- **Relevance justification** → `skills_skill_relevance_justification`
+
+If a skill is not found in the analysis results, say so explicitly rather than guessing or inferring a negative impact.

policies/policies.json

@@ -0,0 +1,203 @@
+[
+  {
+    "type": "playbook",
+    "id": "playbook_api_capability_boundaries",
+    "name": "API Capability Boundaries",
+    "description": "Playbook for api capability boundaries questions",
+    "triggers": [
+      {
+        "type": "keyword",
+        "value": [
+          "job description",
+          "optimize posting",
+          "optimise posting",
+          "rewrite job",
+          "time-to-fill",
+          "time to fill",
+          "SLA deadline",
+          "SLA countdown",
+          "within 30 days",
+          "France",
+          "geography",
+          "geographic",
+          "internal posting",
+          "funnel stage",
+          "days in stage",
+          "time-in-status",
+          "job-card",
+          "job card details",
+          "hiring manager",
+          "list all requisitions",
+          "search requisitions"
+        ],
+        "target": "intent",
+        "case_sensitive": false,
+        "operator": "or"
+      },
+      {
+        "type": "natural_language",
+        "value": [
+          "user asks about data the APIs cannot provide",
+          "user wants to optimize or rewrite a job description",
+          "user asks about time-to-fill or days to fill a role",
+          "user asks about geographic or location filtering",
+          "user wants to list or search across all requisitions",
+          "user asks for funnel stage timing or candidate counts per stage",
+          "user asks for live requisition status or SLA deadlines",
+          "user asks for full job details like title location or manager"
+        ],
+        "target": "intent",
+        "threshold": 0.65
+      }
+    ],
+    "markdown_content": "# API Capability Boundaries\n\nBefore answering any question, verify that the available APIs can actually provide the needed data.\nIf they cannot, tell the user directly — do NOT attempt to cobble together an answer from unrelated endpoints, and do NOT ask for a requisition ID when the query is fundamentally unsupported.\n\n## What the APIs CAN do\n\nThe available tool suite covers two domains:\n\n### Candidate Source Analytics\n- SLA percentage per sourcing channel\n- Total hires per sourcing channel\n- Candidate volume and share per sourcing channel\n- Funnel conversion rates (review %, interview %, offer acceptance %) per source\n- Composite source recommendation summary\n- Metadata: data timeframe, last update date, number of similar requisitions analysed\n- Definitions and methodology: metric definitions, total requisition count used for computation, ML models involved\n\n### Skills Analytics\n- Skill-level statistical analysis (historical counts, SLA correlation)\n- Skill impact on fill rate\n- Skill impact on SLA (delta with/without the skill)\n- Skill relevance justification\n- Data sources and ML models used for recommendations\n- Successful posting criteria and benchmarks\n\n## What the APIs CANNOT do\n\nThe following capabilities are NOT available through any API. If the user asks for any of these, explain that the current API suite does not support it:\n\n- **Job description text**: No API returns or accepts raw job description content. You cannot read, optimise, or rewrite a job description.\n- **Time-to-fill metrics**: No API provides time-to-fill data, whether overall or broken down by source.\n- **Geographic or channel filtering**: No API supports filtering by country, region, or posting channel (internal vs external).\n- **Live requisition status or SLA countdowns**: The APIs provide historical/aggregate analytics, not real-time status tracking or deadline monitoring.\n- **Stage-by-stage funnel timing**: No API returns average days spent in each pipeline stage or candidate counts per stage over time.\n- **Full job-card details**: No API returns comprehensive requisition details like title, location, hiring-manager name, or contact information. The APIs focus on aggregate analytics, not individual job metadata.\n- **Cross-requisition listing or search**: The APIs analyse one requisition at a time against historical data. They cannot list, search, or filter across all open requisitions.\n\n## How to respond when a query is out of scope\n\nWhen you determine that a question cannot be answered with the available APIs:\n\n1. State clearly that the current APIs do not provide the requested data\n2. Be specific about what is missing (e.g., \"the APIs don't expose time-to-fill broken down by source\")\n3. Do NOT ask for a requisition ID — providing one would not help\n4. Do NOT call any API tools — the answer is that the data is unavailable\n5. Do NOT fabricate or infer data that no API returned\n",
+    "steps": [],
+    "priority": 90,
+    "enabled": true
+  },
+  {
+    "type": "playbook",
+    "id": "playbook_multi_api_reasoning",
+    "name": "Multi-API Reasoning",
+    "description": "Playbook for multi-api reasoning questions",
+    "triggers": [
+      {
+        "type": "keyword",
+        "value": [
+          "which sources",
+          "recommend sources",
+          "best sources",
+          "compare sources",
+          "prioritize",
+          "effectiveness",
+          "conversion rate",
+          "multiple metrics",
+          "how many requisitions",
+          "sample size",
+          "most candidates",
+          "how effective"
+        ],
+        "target": "intent",
+        "case_sensitive": false,
+        "operator": "or"
+      },
+      {
+        "type": "natural_language",
+        "value": [
+          "user asks to compare or recommend sourcing channels",
+          "user asks about multiple performance metrics for sources",
+          "user asks how many requisitions were used to compute metrics",
+          "user asks which sources provided the most candidates and their conversion"
+        ],
+        "target": "intent",
+        "threshold": 0.65
+      }
+    ],
+    "markdown_content": "# Multi-API Reasoning\n\nWhen a question asks about multiple dimensions of performance, call the individual specific APIs rather than relying on a single summary endpoint. Summary tools give a quick overview but often lack the granular detail needed for a complete answer.\n\n## Source Performance Questions\n\nWhen comparing or recommending sources across multiple metrics (SLA, volume, conversion, hires), call the specific tools:\n\n- **SLA performance** → `candidate_source_sla_per_source`\n- **Candidate volume and share** → `candidate_source_candidate_volume_by_source`\n- **Funnel conversion rates** (review %, interview %, offer acceptance %) → `candidate_source_funnel_conversion_by_source`\n- **Total hires by source** → `candidate_source_total_hires_by_source`\n\nDo NOT rely solely on `candidate_source_source_recommendation_summary` when the question asks for specific metrics like SLA percentages, offer acceptance rates, or conversion rates. The summary tool is useful for a quick recommendation but does not contain all granular metrics.\n\n## Requisition Count and Sample Size Questions\n\nThere are two different \"counts\" in the system — do not confuse them:\n\n- **\"How many requisitions were used to compute these metrics\"** or **\"sample size\"** → use `candidate_source_definitions_and_methodology`, which returns the total number of requisitions used for computation across the system\n- **\"How many similar requisitions were analysed\"** → use `candidate_source_metadata_and_timeframe`, which returns the count of requisitions similar to the given one\n\nThese return different numbers. Read the question carefully to determine which one is being asked for.\n\n## Skill Analysis Questions\n\nWhen a question asks about skill impact across multiple dimensions (SLA, fill rate, relevance):\n\n- **SLA impact** → `skills_skill_impact_sla`\n- **Fill rate impact** → `skills_skill_impact_fill_rate`\n- **Historical effectiveness and statistical analysis** → `skills_skill_analysis`\n- **Relevance justification** → `skills_skill_relevance_justification`\n\nIf a skill is not found in the analysis results, say so explicitly rather than guessing or inferring a negative impact.\n",
+    "steps": [],
+    "priority": 80,
+    "enabled": true
+  },
+  {
+    "type": "playbook",
+    "id": "playbook_average_vs_total",
+    "name": "Average vs Total Calculations",
+    "description": "Playbook for average vs total calculations questions",
+    "triggers": [
+      {
+        "type": "keyword",
+        "value": [
+          "average",
+          "usually",
+          "on average",
+          "typically",
+          "per posting",
+          "per requisition",
+          "how many candidates"
+        ],
+        "target": "intent",
+        "case_sensitive": false,
+        "operator": "or"
+      },
+      {
+        "type": "natural_language",
+        "value": [
+          "user asks for average or typical values per posting",
+          "user asks how many candidates a posting usually gets"
+        ],
+        "target": "intent",
+        "threshold": 0.65
+      }
+    ],
+    "markdown_content": "# Average vs Total Calculations\n\nWhen the user asks for \"average\", \"typical\", \"usually\", or \"per posting\" values, you must compute an average — do not return a raw total.\n\n## How to compute averages\n\n1. Get the total metric value from the relevant API (e.g., total candidate volume from `candidate_source_candidate_volume_by_source`)\n2. Get the number of similar requisitions from `candidate_source_metadata_and_timeframe`\n3. Divide the total by the number of similar requisitions to get the per-requisition average\n4. Report the average, not the total\n\n## Example\n\nIf the user asks \"How many candidates do we usually get for postings similar to X?\":\n- Total candidates across all sources = 2913\n- Number of similar requisitions = 40\n- Average = 2913 / 40 = ~73 candidates per posting\n- Report: \"On average, similar postings attract 73 candidates\"\n\nDo NOT report 2913 as the answer — that is the total, not the average.\n",
+    "steps": [],
+    "priority": 70,
+    "enabled": true
+  },
+  {
+    "type": "playbook",
+    "id": "playbook_missing_req_id_vs_unsupported",
+    "name": "Missing Requisition ID vs Unsupported Query",
+    "description": "Playbook for missing requisition id vs unsupported query questions",
+    "triggers": [
+      {
+        "type": "keyword",
+        "value": [
+          "this job",
+          "this role",
+          "the position",
+          "for postings",
+          "for roles"
+        ],
+        "target": "intent",
+        "case_sensitive": false,
+        "operator": "or"
+      },
+      {
+        "type": "natural_language",
+        "value": [
+          "user asks a question without specifying a requisition ID",
+          "user refers to a job or role without giving a specific ID"
+        ],
+        "target": "intent",
+        "threshold": 0.6
+      }
+    ],
+    "markdown_content": "# Missing Requisition ID vs Unsupported Query\n\nWhen a question does not include a requisition ID, determine whether providing one would actually help before asking for it.\n\n## Ask for a requisition ID when:\n\nThe question is about something the APIs support but needs a specific requisition to look up:\n- SLA performance by source\n- Candidate volume or hires by source\n- Skill analysis or skill impact\n- Funnel conversion rates\n- Data sources or methodology used\n- Metadata and timeframe\n\nThese all require a requisition ID to return useful results.\n\n## Do NOT ask for a requisition ID when:\n\nThe question is about something no API supports regardless of requisition ID:\n- Job description text (reading, optimizing, rewriting)\n- Time-to-fill metrics (overall or by source)\n- Geographic or location-based filtering\n- Live requisition status or SLA deadline countdowns\n- Stage-by-stage funnel timing (days in each stage)\n- Full job-card details (title, location, hiring-manager info)\n- Listing or searching across all open requisitions\n\nFor these, explain directly that the current APIs do not support the request. Asking for a requisition ID would be misleading because providing one would not help.\n",
+    "steps": [],
+    "priority": 85,
+    "enabled": true
+  },
+  {
+    "type": "tool_guide",
+    "id": "tool_guide_error_tool_warnings",
+    "name": "Error-Prone Tool Warnings",
+    "description": "Tool guide for error-prone endpoints",
+    "triggers": [],
+    "target_tools": [
+      "skills_skill_summary",
+      "candidate_source_source_sla_score",
+      "candidate_source_inactive_sources",
+      "candidate_source_candidate_pipeline_status",
+      "candidate_source_source_sla_check",
+      "candidate_source_funnel_status",
+      "candidate_source_bulk_source_data",
+      "skills_model_registry",
+      "skills_skill_lookup",
+      "candidate_source_source_metrics_lite",
+      "candidate_source_volume_report",
+      "candidate_source_full_candidate_details",
+      "candidate_source_source_directory",
+      "skills_skill_deep_analysis",
+      "candidate_source_sla_extended",
+      "skills_analyze_skill_match",
+      "candidate_source_requisition_details",
+      "candidate_source_list_all_sources",
+      "candidate_source_batch_metrics"
+    ],
+    "guide_content": "# Error-Prone Tool Warning\n\nWARNING: This tool is known to be unreliable. It may return HTTP errors (e.g. 503 Service Unavailable), schema violations, type mismatches, or unexpected data formats.\n\nBefore using this tool, check whether one of the 13 core reliable tools can answer the question instead:\n\n**Reliable Candidate Source tools:** candidate_source_sla_per_source, candidate_source_total_hires_by_source, candidate_source_candidate_volume_by_source, candidate_source_funnel_conversion_by_source, candidate_source_metadata_and_timeframe, candidate_source_definitions_and_methodology, candidate_source_source_recommendation_summary\n\n**Reliable Skills tools:** skills_skill_analysis, skills_skill_impact_fill_rate, skills_skill_impact_sla, skills_skill_relevance_justification, skills_successful_posting_criteria, skills_data_sources_used\n\nIf this tool returns an error or unexpected data:\n- Do NOT report the raw error message to the user\n- Do NOT retry the same tool\n- Check if a reliable tool can provide the needed data\n- If no reliable tool can help, tell the user the data is not available through the current APIs\n",
+    "prepend": true,
+    "priority": 50,
+    "enabled": true
+  }
+]