LoveJesus commited on
Commit
3f6d920
·
verified ·
1 Parent(s): 6b90dfa

Add evaluation metrics to model card (accuracy@0.5 0.9988, ROC AUC 1.0, gap 0.9482)

Browse files
Files changed (1) hide show
  1. README.md +35 -0
README.md CHANGED
@@ -11,6 +11,25 @@ license: mit
11
  datasets:
12
  - LoveJesus/biblical-embedding-dataset-chirho
13
  pipeline_tag: sentence-similarity
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  ---
15
 
16
  <!-- For God so loved the world that he gave his only begotten Son, -->
@@ -54,5 +73,21 @@ print(similarities)
54
 
55
  This is model 5 of 5 in the [bible.systems](https://bible.systems) ML pipeline.
56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
  ---
58
  *For God so loved the world...* — John 3:16
 
11
  datasets:
12
  - LoveJesus/biblical-embedding-dataset-chirho
13
  pipeline_tag: sentence-similarity
14
+ model-index:
15
+ - name: biblical-cross-translation-chirho
16
+ results:
17
+ - task:
18
+ type: sentence-similarity
19
+ name: Cross-Translation Semantic Similarity
20
+ dataset:
21
+ type: LoveJesus/biblical-embedding-dataset-chirho
22
+ name: Biblical Embedding Dataset (Chirho)
23
+ metrics:
24
+ - type: accuracy
25
+ value: 0.9988
26
+ name: Accuracy@0.5
27
+ - type: roc_auc
28
+ value: 1.0000
29
+ name: ROC AUC
30
+ - type: spearmanr
31
+ value: 0.4915
32
+ name: Spearman Correlation
33
  ---
34
 
35
  <!-- For God so loved the world that he gave his only begotten Son, -->
 
73
 
74
  This is model 5 of 5 in the [bible.systems](https://bible.systems) ML pipeline.
75
 
76
+ ## Evaluation Results
77
+
78
+ Evaluated on a held-out test set of cross-translation verse pairs.
79
+
80
+ | Metric | Score |
81
+ |--------|-------|
82
+ | **Accuracy@0.5** (cosine sim threshold) | **0.9988** |
83
+ | **ROC AUC** | **1.0000** |
84
+ | **Spearman Correlation** | **0.4915** |
85
+ | **Avg Positive Similarity** | 0.9841 |
86
+ | **Avg Negative Similarity** | 0.0359 |
87
+ | **Similarity Gap** (pos - neg) | **0.9482** |
88
+
89
+ > The model achieves near-perfect discrimination between same-verse pairs across translations (high positive similarity) and different-verse pairs (low negative similarity), with a gap of 0.95. The Spearman correlation is moderate because within-class similarity variance is low (most positive pairs cluster near 0.98).
90
+
91
+
92
  ---
93
  *For God so loved the world...* — John 3:16