Upload output.log
Browse files- .eval_results/output.log +89 -0
.eval_results/output.log
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
(torch2.7) cccxc584[/dccstor/saon1/open_asr_leaderboard/granite]$ ./run_speculative_bpe.sh
|
| 2 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 3 |
+
backends.update(_get_backends("networkx.backends"))
|
| 4 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 5 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:04<00:00, 1.61s/it]
|
| 6 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1842/1842 [02:32<00:00, 12.11 examples/s]
|
| 7 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1842/1842 [00:00<00:00, 15900.99it/s]
|
| 8 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_voxpopuli_test.jsonl
|
| 9 |
+
ibm-granite/granite-speech-4.1-2b - WER: 5.7%, RTFx: 153.24
|
| 10 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 11 |
+
backends.update(_get_backends("networkx.backends"))
|
| 12 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 13 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:04<00:00, 1.61s/it]
|
| 14 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 11653/11653 [03:02<00:00, 63.96 examples/s]
|
| 15 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 11653/11653 [00:01<00:00, 10828.90it/s]
|
| 16 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_ami_test.jsonl
|
| 17 |
+
ibm-granite/granite-speech-4.1-2b - WER: 8.09%, RTFx: 228.24
|
| 18 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 19 |
+
backends.update(_get_backends("networkx.backends"))
|
| 20 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 21 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:04<00:00, 1.60s/it]
|
| 22 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2737/2737 [02:39<00:00, 17.15 examples/s]
|
| 23 |
+
Samples: 100%|ββββββββββββββββββββοΏ½οΏ½ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2737/2737 [00:00<00:00, 15675.33it/s]
|
| 24 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_earnings22_test.jsonl
|
| 25 |
+
ibm-granite/granite-speech-4.1-2b - WER: 8.37%, RTFx: 166.76
|
| 26 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 27 |
+
backends.update(_get_backends("networkx.backends"))
|
| 28 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 29 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:06<00:00, 2.20s/it]
|
| 30 |
+
Resolving data files: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19/19 [00:00<00:00, 249817.48it/s]
|
| 31 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19898/19898 [08:50<00:00, 37.50 examples/s]
|
| 32 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19898/19898 [00:01<00:00, 13058.22it/s]
|
| 33 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_gigaspeech_test.jsonl
|
| 34 |
+
ibm-granite/granite-speech-4.1-2b - WER: 9.8%, RTFx: 278.09
|
| 35 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 36 |
+
backends.update(_get_backends("networkx.backends"))
|
| 37 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 38 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:05<00:00, 1.71s/it]
|
| 39 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2620/2620 [01:43<00:00, 25.28 examples/s]
|
| 40 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2620/2620 [00:00<00:00, 15887.22it/s]
|
| 41 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_librispeech_test.clean.jsonl
|
| 42 |
+
ibm-granite/granite-speech-4.1-2b - WER: 1.33%, RTFx: 291.5
|
| 43 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 44 |
+
backends.update(_get_backends("networkx.backends"))
|
| 45 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 46 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:05<00:00, 1.72s/it]
|
| 47 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2939/2939 [01:51<00:00, 26.25 examples/s]
|
| 48 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2939/2939 [00:00<00:00, 16532.05it/s]
|
| 49 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_librispeech_test.other.jsonl
|
| 50 |
+
ibm-granite/granite-speech-4.1-2b - WER: 2.5%, RTFx: 254.47
|
| 51 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 52 |
+
backends.update(_get_backends("networkx.backends"))
|
| 53 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 54 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:05<00:00, 1.73s/it]
|
| 55 |
+
Resolving data files: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 38/38 [00:00<00:00, 398458.88it/s]
|
| 56 |
+
Loading dataset shards: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 23/23 [00:03<00:00, 7.09it/s]
|
| 57 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 39341/39341 [23:03<00:00, 28.44 examples/s]
|
| 58 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 39341/39341 [00:02<00:00, 14595.15it/s]
|
| 59 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_spgispeech_test.jsonl
|
| 60 |
+
ibm-granite/granite-speech-4.1-2b - WER: 3.78%, RTFx: 291.15
|
| 61 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 62 |
+
backends.update(_get_backends("networkx.backends"))
|
| 63 |
+
`torch_dtype` is deprecated! Use `dtype` instead!
|
| 64 |
+
Loading checkpoint shards: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 3/3 [00:04<00:00, 1.62s/it]
|
| 65 |
+
Processing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1155/1155 [01:22<00:00, 13.96 examples/s]
|
| 66 |
+
Samples: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1155/1155 [00:00<00:00, 16153.24it/s]
|
| 67 |
+
Results saved at: /dccstor/saon1/open_asr_leaderboard/granite/results/MODEL_ibm-granite-granite-speech-4.1-2b_DATASET_hf-audio-esb-datasets-test-only-sorted_tedlium_test.jsonl
|
| 68 |
+
ibm-granite/granite-speech-4.1-2b - WER: 3.07%, RTFx: 193.34
|
| 69 |
+
/dccstor/saon1/torch2.7/lib64/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
|
| 70 |
+
backends.update(_get_backends("networkx.backends"))
|
| 71 |
+
Filtering models by id: ibm-granite/granite-speech-4.1-2b
|
| 72 |
+
********************************************************************************
|
| 73 |
+
Results per dataset:
|
| 74 |
+
********************************************************************************
|
| 75 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_ami_test: WER = 8.09 %, RTFx = 228.24
|
| 76 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_earnings22_test: WER = 8.37 %, RTFx = 166.76
|
| 77 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_gigaspeech_test: WER = 9.80 %, RTFx = 278.09
|
| 78 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_librispeech_test.clea: WER = 1.33 %, RTFx = 291.50
|
| 79 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_librispeech_test.other: WER = 2.50 %, RTFx = 254.47
|
| 80 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_spgispeech_test: WER = 3.78 %, RTFx = 291.15
|
| 81 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_tedlium_test: WER = 3.07 %, RTFx = 193.34
|
| 82 |
+
ibm/granite-granite-speech-4.1-2b | hf-audio-esb-datasets-test-only-sorted_voxpopuli_test: WER = 5.70 %, RTFx = 153.24
|
| 83 |
+
|
| 84 |
+
********************************************************************************
|
| 85 |
+
Composite Results:
|
| 86 |
+
********************************************************************************
|
| 87 |
+
ibm/granite-granite-speech-4.1-2b: WER = 5.33 %
|
| 88 |
+
ibm/granite-granite-speech-4.1-2b: RTFx = 267.84
|
| 89 |
+
********************************************************************************
|