# Pod Manager Log

## Environment
- GPUs: 0x ?, ? MB each
- Persistent storage: /workspace

## Current Status
- Uptime: 18h14m
- Cost: $2.49/hr, ~$45.43 so far

## Results
| Trial | Strategy | Params | val_loss | Acc | Status | Notes |
|-------|----------|--------|----------|-----|--------|-------|
| 0 | bottleneck | 262,144 | 2.6815 | 25.4% | completed | MCP smoke test: 256k bottleneck, 100 steps, MATH+compile+fp16 |
| 1 | bottleneck | ? | --- | --- | killed |  |
| 2 | bottleneck | ? | --- | --- | killed |  |
| 3 | bottleneck | ? | --- | --- | killed |  |
| 4 | bottleneck | ? | --- | --- | killed |  |
| 5 | bottleneck | ? | --- | --- | killed |  |
| 6 | bottleneck | ? | --- | --- | completed |  |
| 7 | bottleneck | ? | --- | --- | completed |  |
| 8 | bottleneck | ? | --- | --- | completed |  |
| 9 | bottleneck | ? | --- | --- | killed |  |
| 10 | bottleneck | ? | --- | --- | completed |  |

## Recent Events
- [2026-04-05T01:27:00] trial_completed (trial 7) {"best_val_loss": null, "best_accuracy": null, "param_count": null, "steps": 0}
- [2026-04-05T01:27:00] gpu_idle {"message": "All GPUs are idle"}
- [2026-04-05T01:27:20] trial_started (trial 8) {"strategy": "bottleneck", "gpu": 0, "config": {"elo_min": 1800, "elo_max": 1900, "max_games": 30000000, "val_games": 5000, "min_ply": 10, "total_steps": 50, "batch_size": 256, "lr": 0.0003, "weight_decay": 0.0, "warmup_frac": 0.02, "warmup_steps": null, "max_grad_norm": 1.0, "patience": 9999, "eval_interval": 2500, "log_interval": 10, "pause_after_steps": null, "mate_boost": 0.0, "no_outcome_token": false, "discard_ply_limit": false, "amp_dtype": "bfloat16", "no_compile": true, "sdpa_math": true, "num_workers": 2, "device": "cuda", "log_dir": "/workspace/logs", "hf_repo": null, "local_checkpoints": true, "resume": null, "wandb": false, "cache_dir": "/dev/shm/pawn_cache", "run_type": "adapter", "strategy": "bottleneck", "checkpoint": "thomas-schweich/pawn-base", "pgn": "thomas-schweich/pawn-lichess-full", "adapter_layers": "4,5,6,7", "bottleneck_dim": 2440, "no_adapt_attn": false, "no_adapt_ffn": false, "lora_rank": null, "lora_targets": null, "lora_ffn": false, "density": null, "sparse_targets": null, "sparse_ffn": false, "use_output_film": false, "rosa_mode": null, "rosa_warmup_steps": 128, "mask_samples": 32, "grad_alpha": 2, "unfreeze_layers": null, "d_model": null, "n_layers": null, "n_heads": null, "epochs": 9999, "val_every": 9999}}
- [2026-04-05T01:27:55] trial_completed (trial 8) {"best_val_loss": null, "best_accuracy": null, "param_count": null, "steps": 0}
- [2026-04-05T01:27:55] gpu_idle {"message": "All GPUs are idle"}
- [2026-04-05T01:29:33] trial_started (trial 9) {"strategy": "bottleneck", "gpu": 0, "config": {"elo_min": 1800, "elo_max": 1900, "max_games": 30000000, "val_games": 50000, "min_ply": 10, "total_steps": 200000, "batch_size": 256, "lr": 0.0003, "weight_decay": 0.0, "warmup_frac": 0.02, "warmup_steps": null, "max_grad_norm": 1.0, "patience": 9999, "eval_interval": 2500, "log_interval": 10, "pause_after_steps": 15000, "mate_boost": 0.0, "no_outcome_token": false, "discard_ply_limit": false, "amp_dtype": "bfloat16", "no_compile": true, "sdpa_math": true, "num_workers": 2, "device": "cuda", "log_dir": "/workspace/logs", "hf_repo": null, "local_checkpoints": true, "resume": null, "wandb": false, "cache_dir": "/dev/shm/pawn_cache", "run_type": "adapter", "strategy": "bottleneck", "checkpoint": "thomas-schweich/pawn-base", "pgn": "thomas-schweich/pawn-lichess-full", "adapter_layers": "4,5,6,7", "bottleneck_dim": 2440, "no_adapt_attn": false, "no_adapt_ffn": false, "lora_rank": null, "lora_targets": null, "lora_ffn": false, "density": null, "sparse_targets": null, "sparse_ffn": false, "use_output_film": false, "rosa_mode": null, "rosa_warmup_steps": 128, "mask_samples": 32, "grad_alpha": 2, "unfreeze_layers": null, "d_model": null, "n_layers": null, "n_heads": null, "epochs": 9999, "val_every": 9999}}
- [2026-04-05T01:33:37] trial_killed (trial 9) {}
- [2026-04-05T01:34:21] trial_started (trial 10) {"strategy": "bottleneck", "gpu": 0, "config": {"elo_min": 1800, "elo_max": 1900, "max_games": 30000000, "val_games": 50000, "min_ply": 10, "total_steps": 200000, "batch_size": 256, "lr": 0.0003, "weight_decay": 0.0, "warmup_frac": 0.02, "warmup_steps": null, "max_grad_norm": 1.0, "patience": 9999, "eval_interval": 2500, "log_interval": 100, "pause_after_steps": 15000, "mate_boost": 0.0, "no_outcome_token": false, "discard_ply_limit": false, "amp_dtype": "bfloat16", "no_compile": true, "sdpa_math": true, "num_workers": 2, "device": "cuda", "log_dir": "/workspace/logs", "hf_repo": null, "local_checkpoints": true, "resume": null, "wandb": false, "cache_dir": "/dev/shm/pawn_cache", "run_type": "adapter", "strategy": "bottleneck", "checkpoint": "thomas-schweich/pawn-base", "pgn": "thomas-schweich/pawn-lichess-full", "adapter_layers": "4,5,6,7", "bottleneck_dim": 2440, "no_adapt_attn": false, "no_adapt_ffn": false, "lora_rank": null, "lora_targets": null, "lora_ffn": false, "density": null, "sparse_targets": null, "sparse_ffn": false, "use_output_film": false, "rosa_mode": null, "rosa_warmup_steps": 128, "mask_samples": 32, "grad_alpha": 2, "unfreeze_layers": null, "d_model": null, "n_layers": null, "n_heads": null, "epochs": 9999, "val_every": 9999}}
- [2026-04-05T02:30:43] trial_completed (trial 10) {"best_val_loss": null, "best_accuracy": null, "param_count": null, "steps": 0}
- [2026-04-05T02:30:43] gpu_idle {"message": "All GPUs are idle"}