Report for nateraw/bert-base-uncased-emotion
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 2 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset dair-ai/emotion (subset split, split validation).
You can find a full version of scan report here.
👉Performance issues (1)
For records in the dataset where text contains "know", the Precision is 6.45% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text contains "know" |
Precision = 0.876 | -6.45% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 17 | i know what it feels like he stressed glaring down at her as she squeezed more soap onto her sponge | anger | sadness (p = 0.89) |
| 91 | i feel like the people i know are really generous and i have my needs met | joy | love (p = 0.68) |
| 164 | i have stayed at heritage christian because of the fulfillment that i feel in doing christ s work in action by being the hands the eyes the legs and the voice of supporting the individuals that i have been blessed to know and support | joy | love (p = 0.61) |
👉Robustness issues (1)
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 21.3% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| major 🔴 | Fail rate = 0.213 | Add typos | 213/1000 tested samples (21.3%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 656 | i feel a little bit more nostalgic when those memories come to mind | i feel a little bit more nosftalic when those memories comwe to mind | love (p = 0.99) | joy (p = 0.94) |
| 734 | i can talk to her about almost anything i want to and she just listens and she doesnt make me feel like a whiney brat and she helps me sort my thoughts and make decisions while keeping me where she feels im safe | i can talk to her about almost anything i want to and she just lisrens and she doesnt make me feel liek a shiney brat and she helps me sort my thoughts and make decisions while keeping me where she fes im safe | sadness (p = 0.99) | joy (p = 0.99) |
| 1403 | i feel the need to preface this by saying that i am strongly in favor of keeping violent or otherwise inappropriate videogames out of the hands of minors and i believe that this is an issue that parents and the government need to work on together | i feel the need to preface this by saying that i am ateongly in faor of keeping volent or otherwise inappropriate videogames outo f yhe hands of minor san di believe that this is an issue that parents and the government need to work on tovether | anger (p = 0.97) | sadness (p = 0.88) |
We've generated test suites according to your scan results! Checkout the Test Suite in our Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.