Upload logs/nanochat-tr-d20-bpe32k-492421.err

logs/nanochat-tr-d20-bpe32k-492421.err

@@ -0,0 +1,133 @@
+	Bu module sadece a100q ve a100x4q kuyrugundaki makinalarda calisir.
+2026-06-07 16:23:31,528 - nanochat.common - INFO - Distributed world size: 4
+wandb: Currently logged in as: nurcunal (nurcunal-bogaziciuniversitesi) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run tr-d20-bpe32k-20260607-162301
+wandb: Tracking run with wandb version 0.22.3
+wandb: Run data is saved locally in /ari/users/nunal/nanochat-turk-d20-bpe32k/wandb/wandb/run-20260607_162332-tr-d20-bpe32k-20260607-162301
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Syncing run tr-d20-bpe32k
+wandb: ⭐️ View project at https://wandb.ai/nurcunal-bogaziciuniversitesi/nanochat-turk
+wandb: 🚀 View run at https://wandb.ai/nurcunal-bogaziciuniversitesi/nanochat-turk/runs/tr-d20-bpe32k-20260607-162301
+2026-06-07 19:31:02,986 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_002500_rank2.pt
+2026-06-07 19:31:03,721 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_002500_rank3.pt
+2026-06-07 19:31:03,723 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_002500_rank1.pt
+2026-06-07 19:31:04,669 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_002500.pt
+2026-06-07 19:31:04,672 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_002500.json
+2026-06-07 19:31:05,600 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_002500_rank0.pt
+2026-06-07 22:36:52,330 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_005000_rank1.pt
+2026-06-07 22:36:52,505 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_005000_rank2.pt
+2026-06-07 22:36:52,614 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_005000_rank3.pt
+2026-06-07 22:37:05,783 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_005000.pt
+2026-06-07 22:37:05,790 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_005000.json
+2026-06-07 22:37:07,259 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_005000_rank0.pt
+2026-06-08 01:42:55,280 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_007500_rank2.pt
+2026-06-08 01:42:55,666 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_007500_rank1.pt
+2026-06-08 01:42:55,769 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_007500_rank3.pt
+2026-06-08 01:42:58,578 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_007500.pt
+2026-06-08 01:42:58,582 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_007500.json
+2026-06-08 01:43:00,074 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_007500_rank0.pt
+2026-06-08 04:48:53,363 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_010000_rank2.pt
+2026-06-08 04:48:53,534 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_010000_rank1.pt
+2026-06-08 04:48:56,328 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_010000_rank3.pt
+2026-06-08 04:48:56,685 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_010000.pt
+2026-06-08 04:48:56,689 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_010000.json
+2026-06-08 04:49:01,264 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_010000_rank0.pt
+2026-06-08 07:54:39,968 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_012500_rank3.pt
+2026-06-08 07:54:39,991 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_012500_rank2.pt
+2026-06-08 07:54:40,054 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_012500_rank1.pt
+2026-06-08 07:54:41,707 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_012500.pt
+2026-06-08 07:54:41,710 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_012500.json
+2026-06-08 07:54:42,841 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_012500_rank0.pt
+2026-06-08 11:00:33,210 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_015000_rank1.pt
+2026-06-08 11:00:33,211 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_015000_rank2.pt
+2026-06-08 11:00:33,637 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_015000_rank3.pt
+2026-06-08 11:00:47,652 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_015000.pt
+2026-06-08 11:00:47,655 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_015000.json
+2026-06-08 11:00:53,283 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_015000_rank0.pt
+2026-06-08 13:37:24,481 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_017100_rank2.pt
+2026-06-08 13:37:24,556 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_017100_rank1.pt
+2026-06-08 13:37:24,815 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_017100_rank3.pt
+2026-06-08 13:37:28,031 - nanochat.checkpoint_manager - INFO - Saved model parameters to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/model_017100.pt
+2026-06-08 13:37:28,038 - nanochat.checkpoint_manager - INFO - Saved metadata to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/meta_017100.json
+2026-06-08 13:37:29,349 - nanochat.checkpoint_manager - INFO - Saved optimizer state to: /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20/optim_017100_rank0.pt
+wandb: updating run metadata
+wandb: uploading output.log; uploading wandb-summary.json
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading history steps 189-189, summary, console lines 17323-17336
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
+wandb: uploading wandb-summary.json; uploading config.yaml
+wandb: uploading wandb-summary.json; uploading config.yaml; uploading data
+wandb: uploading wandb-summary.json; uploading config.yaml
+wandb: uploading config.yaml
+wandb: uploading data
+wandb: 
+wandb: Run history:
+wandb:                 step ▁▁▁▁▁▂▂▂▂▂▃▃▄▄▄▄▄▄▄▄▅▅▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇███
+wandb: total_training_flops ▁▁▁▁▂▂▂▂▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇█████
+wandb:  total_training_time ▁▁▁▂▂▂▂▂▃▃▃▃▃▃▃▄▄▄▄▄▄▄▄▅▅▅▅▆▆▆▆▇▇▇▇▇▇███
+wandb:             train/dt ▅▃▄▅▃▃▃▁▁▃▃▄▄▁▄▃▅▆▄▄▄▃▄▃▃▃▃▃▃▃▄▄▃▁▄█▄▃▃▃
+wandb:           train/loss █▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▁▁▁▂▂▂▁▁▁▁▂▂▁▁▁▁▁▁▁▁
+wandb:            train/lrm ███████████▇▇▇▇▆▆▆▆▆▅▅▅▄▄▄▄▄▃▃▃▃▃▃▂▂▁▁▁▁
+wandb:            train/mfu ▁███████████████████████████████████████
+wandb:    train/tok_per_sec ▁███████████████████████████████████████
+wandb:              val/bpb █▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁
+wandb: 
+wandb: Run summary:
+wandb:                 step 17100
+wandb: total_training_flops 5.809722400073318e+19
+wandb:  total_training_time 76059.47399
+wandb:             train/dt 4.43947
+wandb:          train/epoch 1 pq: 10 rg: 1744
+wandb:           train/loss 2.38038
+wandb:            train/lrm 0.05855
+wandb:            train/mfu 61.32155
+wandb:    train/tok_per_sec 236193
+wandb:              val/bpb 0.62321
+wandb: 
+wandb: 🚀 View run tr-d20-bpe32k at: https://wandb.ai/nurcunal-bogaziciuniversitesi/nanochat-turk/runs/tr-d20-bpe32k-20260607-162301
+wandb: ⭐️ View project at: https://wandb.ai/nurcunal-bogaziciuniversitesi/nanochat-turk
+wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
+wandb: Find logs at: .-d20-bpe32k/wandb/wandb/run-20260607_162332-tr-d20-bpe32k-20260607-162301/logs
+2026-06-08 13:54:06,931 - nanochat.common - INFO - Distributed world size: 4
+2026-06-08 13:54:06,934 - nanochat.checkpoint_manager - INFO - Loading model from /ari/users/nunal/nanochat-turk-d20-bpe32k/base_checkpoints/tr_d20_bpe_32768_chinchilla20 with step 17100
+2026-06-08 13:54:08,704 - nanochat.checkpoint_manager - INFO - Building model with config: {'sequence_len': 2048, 'vocab_size': 32768, 'n_layer': 20, 'n_head': 10, 'n_kv_head': 10, 'n_embd': 1280, 'window_pattern': 'L'}