voidful's picture
End of training
6d8f299 verified
Loading weights: 0%| | 0/1184 [00:00<?, ?it/s] Loading weights: 0%| | 0/1184 [00:00<?, ?it/s] Loading weights: 0%| | 1/1184 [00:00<07:41, 2.56it/s] Loading weights: 0%| | 1/1184 [00:00<07:11, 2.74it/s] Loading weights: 0%| | 2/1184 [00:00<07:45, 2.54it/s] Loading weights: 0%| | 2/1184 [00:00<08:36, 2.29it/s] Loading weights: 1%| | 14/1184 [00:00<00:51, 22.91it/s] Loading weights: 1%| | 14/1184 [00:00<00:58, 20.05it/s] Loading weights: 2%|▏ | 27/1184 [00:00<00:26, 43.50it/s] Loading weights: 2%|▏ | 27/1184 [00:01<00:30, 37.48it/s] Loading weights: 3%|β–Ž | 37/1184 [00:01<00:20, 54.88it/s] Loading weights: 3%|β–Ž | 37/1184 [00:01<00:23, 48.60it/s] Loading weights: 4%|▍ | 46/1184 [00:01<00:20, 56.51it/s] Loading weights: 4%|▍ | 45/1184 [00:01<00:23, 47.91it/s] Loading weights: 5%|▍ | 54/1184 [00:01<00:18, 61.36it/s] Loading weights: 6%|β–Œ | 67/1184 [00:01<00:15, 74.00it/s] Loading weights: 4%|▍ | 52/1184 [00:01<00:23, 48.73it/s] Loading weights: 6%|β–Œ | 66/1184 [00:01<00:16, 66.28it/s] Loading weights: 7%|β–‹ | 80/1184 [00:01<00:13, 83.62it/s] Loading weights: 8%|β–Š | 91/1184 [00:01<00:12, 89.71it/s] Loading weights: 7%|β–‹ | 77/1184 [00:01<00:14, 74.04it/s] Loading weights: 7%|β–‹ | 86/1184 [00:01<00:15, 68.91it/s] Loading weights: 9%|β–Š | 101/1184 [00:01<00:15, 70.62it/s] Loading weights: 8%|β–Š | 95/1184 [00:02<00:15, 69.47it/s] Loading weights: 10%|β–ˆ | 119/1184 [00:02<00:11, 92.04it/s] Loading weights: 11%|β–ˆ | 130/1184 [00:02<00:11, 95.39it/s] Loading weights: 9%|β–Š | 103/1184 [00:02<00:17, 62.62it/s] Loading weights: 12%|β–ˆβ– | 141/1184 [00:02<00:10, 96.35it/s] Loading weights: 10%|β–ˆ | 119/1184 [00:02<00:12, 82.14it/s] Loading weights: 11%|β–ˆ | 132/1184 [00:02<00:11, 92.26it/s] Loading weights: 13%|β–ˆβ–Ž | 152/1184 [00:02<00:12, 83.55it/s] Loading weights: 12%|β–ˆβ– | 143/1184 [00:02<00:11, 93.22it/s] Loading weights: 14%|β–ˆβ–Ž | 162/1184 [00:02<00:11, 86.05it/s] Loading weights: 15%|β–ˆβ– | 173/1184 [00:02<00:11, 90.35it/s] Loading weights: 13%|β–ˆβ–Ž | 153/1184 [00:02<00:14, 71.86it/s] Loading weights: 16%|β–ˆβ–Œ | 186/1184 [00:02<00:10, 96.23it/s] Loading weights: 15%|β–ˆβ– | 172/1184 [00:02<00:10, 94.32it/s] Loading weights: 17%|β–ˆβ–‹ | 197/1184 [00:02<00:10, 97.72it/s] Loading weights: 15%|β–ˆβ–Œ | 183/1184 [00:03<00:10, 97.21it/s] Loading weights: 16%|β–ˆβ–‹ | 194/1184 [00:03<00:10, 97.67it/s] Loading weights: 18%|β–ˆβ–Š | 208/1184 [00:03<00:12, 79.35it/s] Loading weights: 19%|β–ˆβ–‰ | 225/1184 [00:03<00:09, 96.18it/s] Loading weights: 17%|β–ˆβ–‹ | 205/1184 [00:03<00:11, 85.57it/s] Loading weights: 20%|β–ˆβ–ˆ | 237/1184 [00:03<00:09, 99.67it/s] Loading weights: 18%|β–ˆβ–Š | 218/1184 [00:03<00:10, 96.02it/s] Loading weights: 21%|β–ˆβ–ˆ | 249/1184 [00:03<00:09, 101.84it/s] Loading weights: 19%|β–ˆβ–‰ | 229/1184 [00:03<00:10, 89.23it/s] Loading weights: 20%|β–ˆβ–ˆ | 240/1184 [00:03<00:10, 90.57it/s] Loading weights: 22%|β–ˆβ–ˆβ– | 260/1184 [00:03<00:11, 80.58it/s] Loading weights: 21%|β–ˆβ–ˆβ– | 253/1184 [00:03<00:09, 97.23it/s] Loading weights: 23%|β–ˆβ–ˆβ–Ž | 278/1184 [00:03<00:09, 100.54it/s] Loading weights: 25%|β–ˆβ–ˆβ– | 292/1184 [00:03<00:08, 102.25it/s] Loading weights: 22%|β–ˆβ–ˆβ– | 264/1184 [00:03<00:10, 85.16it/s] Loading weights: 26%|β–ˆβ–ˆβ–Œ | 305/1184 [00:03<00:08, 108.18it/s] Loading weights: 23%|β–ˆβ–ˆβ–Ž | 278/1184 [00:04<00:09, 96.00it/s] Loading weights: 25%|β–ˆβ–ˆβ– | 291/1184 [00:04<00:08, 102.40it/s] Loading weights: 27%|β–ˆβ–ˆβ–‹ | 317/1184 [00:04<00:09, 87.03it/s] Loading weights: 26%|β–ˆβ–ˆβ–Œ | 302/1184 [00:04<00:08, 98.45it/s] Loading weights: 28%|β–ˆβ–ˆβ–Š | 332/1184 [00:04<00:08, 94.86it/s] Loading weights: 26%|β–ˆβ–ˆβ–‹ | 313/1184 [00:04<00:11, 78.92it/s] Loading weights: 29%|β–ˆβ–ˆβ–‰ | 346/1184 [00:04<00:08, 98.95it/s] Loading weights: 28%|β–ˆβ–ˆβ–Š | 331/1184 [00:04<00:08, 98.00it/s] Loading weights: 30%|β–ˆβ–ˆβ–ˆ | 359/1184 [00:04<00:07, 104.62it/s] Loading weights: 29%|β–ˆβ–ˆβ–‰ | 342/1184 [00:04<00:08, 99.98it/s] Loading weights: 31%|β–ˆβ–ˆβ–ˆβ– | 371/1184 [00:04<00:08, 92.41it/s] Loading weights: 30%|β–ˆβ–ˆβ–‰ | 353/1184 [00:04<00:08, 97.87it/s] Loading weights: 32%|β–ˆβ–ˆβ–ˆβ– | 384/1184 [00:04<00:07, 100.47it/s] Loading weights: 34%|β–ˆβ–ˆβ–ˆβ–Ž | 397/1184 [00:04<00:07, 106.02it/s] Loading weights: 31%|β–ˆβ–ˆβ–ˆ | 364/1184 [00:04<00:09, 84.79it/s] Loading weights: 32%|β–ˆβ–ˆβ–ˆβ– | 374/1184 [00:05<00:09, 87.07it/s] Loading weights: 35%|β–ˆβ–ˆβ–ˆβ– | 409/1184 [00:05<00:07, 103.56it/s] Loading weights: 33%|β–ˆβ–ˆβ–ˆβ–Ž | 385/1184 [00:05<00:08, 90.78it/s] Loading weights: 35%|β–ˆβ–ˆβ–ˆβ–Œ | 420/1184 [00:05<00:09, 82.31it/s] Loading weights: 34%|β–ˆβ–ˆβ–ˆβ–Ž | 398/1184 [00:05<00:08, 96.61it/s] Loading weights: 37%|β–ˆβ–ˆβ–ˆβ–‹ | 437/1184 [00:05<00:07, 98.98it/s] Loading weights: 35%|β–ˆβ–ˆβ–ˆβ– | 410/1184 [00:05<00:07, 99.72it/s] Loading weights: 38%|β–ˆβ–ˆβ–ˆβ–Š | 448/1184 [00:05<00:07, 99.70it/s] Loading weights: 36%|β–ˆβ–ˆβ–ˆβ–Œ | 421/1184 [00:05<00:09, 82.64it/s] Loading weights: 39%|β–ˆβ–ˆβ–ˆβ–‰ | 459/1184 [00:05<00:07, 101.98it/s] Loading weights: 37%|β–ˆβ–ˆβ–ˆβ–‹ | 437/1184 [00:05<00:07, 96.96it/s] Loading weights: 40%|β–ˆβ–ˆβ–ˆβ–‰ | 470/1184 [00:05<00:08, 87.51it/s] Loading weights: 38%|β–ˆβ–ˆβ–ˆβ–Š | 449/1184 [00:05<00:07, 100.80it/s] Loading weights: 41%|β–ˆβ–ˆβ–ˆβ–ˆ | 480/1184 [00:05<00:07, 90.02it/s] Loading weights: 39%|β–ˆβ–ˆβ–ˆβ–‰ | 461/1184 [00:05<00:07, 102.68it/s] Loading weights: 41%|β–ˆβ–ˆβ–ˆβ–ˆβ– | 491/1184 [00:05<00:07, 87.78it/s] Loading weights: 43%|β–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 505/1184 [00:06<00:06, 98.91it/s] Loading weights: 40%|β–ˆβ–ˆβ–ˆβ–‰ | 472/1184 [00:06<00:08, 79.83it/s] Loading weights: 44%|β–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 517/1184 [00:06<00:06, 102.89it/s] Loading weights: 41%|β–ˆβ–ˆβ–ˆβ–ˆβ– | 490/1184 [00:06<00:07, 98.57it/s] Loading weights: 42%|β–ˆβ–ˆβ–ˆβ–ˆβ– | 501/1184 [00:06<00:06, 101.10it/s] Loading weights: 45%|β–ˆβ–ˆβ–ˆβ–ˆβ– | 528/1184 [00:06<00:07, 89.24it/s] Loading weights: 43%|β–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 512/1184 [00:06<00:06, 102.54it/s] Loading weights: 46%|β–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 543/1184 [00:06<00:06, 101.28it/s] Loading weights: 47%|β–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 556/1184 [00:06<00:05, 106.59it/s] Loading weights: 44%|β–ˆβ–ˆβ–ˆβ–ˆβ– | 523/1184 [00:06<00:07, 87.76it/s] Loading weights: 48%|β–ˆβ–ˆβ–ˆβ–ˆβ–Š | 568/1184 [00:06<00:05, 104.13it/s] Loading weights: 45%|β–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 533/1184 [00:06<00:07, 89.71it/s] Loading weights: 46%|β–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 544/1184 [00:06<00:06, 91.70it/s] Loading weights: 49%|β–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 579/1184 [00:06<00:07, 82.76it/s] Loading weights: 47%|β–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 557/1184 [00:07<00:06, 98.09it/s] Loading weights: 50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 595/1184 [00:07<00:06, 98.01it/s] Loading weights: 48%|β–ˆβ–ˆβ–ˆβ–ˆβ–Š | 568/1184 [00:07<00:06, 101.15it/s] Loading weights: 51%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 609/1184 [00:07<00:05, 106.05it/s] Loading weights: 49%|β–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 579/1184 [00:07<00:07, 80.51it/s] Loading weights: 52%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 621/1184 [00:07<00:05, 102.20it/s] Loading weights: 50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 596/1184 [00:07<00:05, 98.07it/s] Loading weights: 53%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 632/1184 [00:07<00:06, 81.51it/s] Loading weights: 51%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 607/1184 [00:07<00:05, 100.52it/s] Loading weights: 55%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 649/1184 [00:07<00:05, 98.39it/s] Loading weights: 52%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 618/1184 [00:07<00:05, 100.79it/s] Loading weights: 56%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 662/1184 [00:07<00:05, 103.45it/s] Loading weights: 53%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 629/1184 [00:07<00:06, 87.46it/s] Loading weights: 57%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 674/1184 [00:07<00:05, 98.57it/s] Loading weights: 54%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 639/1184 [00:07<00:06, 90.49it/s] Loading weights: 55%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 650/1184 [00:08<00:05, 92.76it/s] Loading weights: 58%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 685/1184 [00:08<00:06, 81.47it/s] Loading weights: 56%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 663/1184 [00:08<00:05, 100.39it/s] Loading weights: 59%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 702/1184 [00:08<00:04, 99.12it/s] Loading weights: 57%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 674/1184 [00:08<00:05, 98.08it/s] Loading weights: 60%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 715/1184 [00:08<00:04, 105.24it/s] Loading weights: 61%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 727/1184 [00:08<00:04, 103.59it/s] Loading weights: 58%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 685/1184 [00:08<00:06, 79.96it/s] Loading weights: 59%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 702/1184 [00:08<00:04, 98.06it/s] Loading weights: 62%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 738/1184 [00:08<00:05, 83.36it/s] Loading weights: 60%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 713/1184 [00:08<00:04, 100.48it/s] Loading weights: 64%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 755/1184 [00:08<00:04, 100.31it/s] Loading weights: 61%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 724/1184 [00:08<00:04, 96.68it/s] Loading weights: 65%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 768/1184 [00:08<00:03, 105.42it/s] Loading weights: 62%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 735/1184 [00:08<00:05, 86.24it/s] Loading weights: 66%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 780/1184 [00:08<00:03, 102.57it/s] Loading weights: 63%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 745/1184 [00:09<00:04, 89.49it/s] Loading weights: 64%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 756/1184 [00:09<00:04, 93.74it/s] Loading weights: 67%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 791/1184 [00:09<00:04, 82.51it/s] Loading weights: 65%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 769/1184 [00:09<00:04, 102.81it/s] Loading weights: 68%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 808/1184 [00:09<00:03, 99.00it/s] Loading weights: 66%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 780/1184 [00:09<00:03, 102.88it/s] Loading weights: 69%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 822/1184 [00:09<00:03, 102.13it/s] Loading weights: 71%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 836/1184 [00:09<00:03, 104.41it/s] Loading weights: 67%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 791/1184 [00:09<00:04, 82.86it/s] Loading weights: 68%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 808/1184 [00:09<00:03, 96.92it/s] Loading weights: 72%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 847/1184 [00:09<00:03, 90.57it/s] Loading weights: 69%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰ | 821/1184 [00:09<00:03, 104.61it/s] Loading weights: 86%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 1013/1184 [00:09<00:00, 449.90it/s] Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1184/1184 [00:09<00:00, 120.37it/s] Loading weights: 70%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 833/1184 [00:09<00:03, 101.81it/s] Loading weights: 71%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 844/1184 [00:10<00:04, 84.70it/s] Loading weights: 83%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 983/1184 [00:10<00:00, 383.93it/s] Loading weights: 99%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‰| 1176/1184 [00:10<00:00, 759.63it/s] Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1184/1184 [00:10<00:00, 114.90it/s]
Installed CUDA version 12.4 does not match the version torch was compiled with 12.8 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.4 does not match the version torch was compiled with 12.8 but since the APIs are compatible, accepting this combination
Writing model shards: 0%| | 0/2 [00:00<?, ?it/s] Writing model shards: 0%| | 0/2 [00:00<?, ?it/s] Writing model shards: 50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 1/2 [00:50<00:50, 50.02s/it] Writing model shards: 50%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆ | 1/2 [00:51<00:51, 51.78s/it] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:55<00:00, 24.05s/it] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:55<00:00, 27.94s/it] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:56<00:00, 24.18s/it] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:56<00:00, 28.32s/it]