Upload README.md with huggingface_hub

README.md

@@ -20,13 +20,13 @@ citation: |
     url = {https://github.com/thomas-schweich/PAWN},
     license = {Apache-2.0}
   }
-model_params: 68376320
+model_params: 66909440
 d_model: 640
 n_layers: 10
 n_heads: 8
 d_ff: 2560
-context_length: 256
-vocab_size: 4284
+context_length: 512
+vocab_size: 1980
 datasets:
   - random-chess-games
 language:
@@ -40,32 +40,31 @@ model-index:
           type: next-token-prediction
           name: Chess Move Prediction (Random Games)
         metrics:
-
+          - name: Game Completion Rate
+            type: accuracy
+            value: 0.997559
           - name: Legal Move Rate
             type: accuracy
-            value: 0.9989
-
+            value: 0.999990
           - name: Top-1 Accuracy
             type: accuracy
-            value: 0.0695
-
+            value: 0.0863
           - name: Top-5 Accuracy
             type: accuracy
-            value: 0.2773
-
+            value: 0.3556
           - name: Val Loss
             type: loss
-            value: 3.0919
-          - name: Games Seen
+            value: 2.8652
+          - name: Total Training Sequences
             type: other
-            value: 25600000
+            value: 51200000
 ---
 
 # PAWN-Large
 
 **PAWN** (Playstyle-Agnostic World-model Network for Chess) is a causal transformer trained on random chess games. It learns legal moves, board state representations, and game dynamics purely from uniformly random legal move sequences -- no strategic play, no hand-crafted features, no external game databases.
 
-This is the **large** variant (~68.4M parameters). PAWN is designed as a frozen backbone for parameter-efficient finetuning into player models with arbitrary playstyles.
+This is the **large** variant (~66.9M parameters). PAWN is designed as a frozen backbone for parameter-efficient finetuning into player models with arbitrary playstyles.
 
 **[GitHub Repository](https://github.com/thomas-schweich/PAWN)** -- full source code, training scripts, adapter implementations, and documentation.
 
@@ -73,32 +72,44 @@ This is the **large** variant (~68.4M parameters). PAWN is designed as a frozen
 
 | Variant | Parameters | Link |
 |---------|------------|------|
-| PAWN-Small | ~9.5M | [thomas-schweich/pawn-small](https://huggingface.co/thomas-schweich/pawn-small) |
-| PAWN (Base) | ~35.8M | [thomas-schweich/pawn-base](https://huggingface.co/thomas-schweich/pawn-base) |
-| PAWN-Large | ~68.4M | [thomas-schweich/pawn-large](https://huggingface.co/thomas-schweich/pawn-large) |
+| PAWN-Small | ~9M | [thomas-schweich/pawn-small](https://huggingface.co/thomas-schweich/pawn-small) |
+| PAWN (Base) | ~35M | [thomas-schweich/pawn-base](https://huggingface.co/thomas-schweich/pawn-base) |
+| PAWN-Large | ~67M | [thomas-schweich/pawn-large](https://huggingface.co/thomas-schweich/pawn-large) |
+
+A previous generation of PAWN backbones (`pawn-{small,base,large}-legacy`) used a 4,278-token coordinate vocabulary, a 256-token context window, and outcome conditioning. They are still available on HuggingFace; see [docs/LEGACY.md](https://github.com/thomas-schweich/PAWN/blob/main/docs/LEGACY.md) for the full story.
 
 ## Headline Metrics
 
+These come from the published `model.safetensors` (step 195,000 out of 200,000 — the best 5,000-step-cadence checkpoint by val loss), measured on a fresh validation set of random games.
+
 | Metric | Value |
 |--------|-------|
-| Legal move rate | 99.89% |
-| Top-1 accuracy | 6.95% |
-| Top-5 accuracy | 27.73% |
-| Val loss | 3.092 |
+| Game completion rate | 99.76% |
+| Per-move legal rate | 99.9990% |
+| Late-game legal rate | 99.9996% |
+| Top-1 accuracy | 8.63% |
+| Top-5 accuracy | 35.56% |
+| Val loss | 2.865 |
+| Val perplexity | 17.55 |
+
+**Game completion rate** is the share of validation games in which *every* prediction along one side's plies was a legal move. The measurement is **non-autoregressive**: at each ply the model is shown the true ground-truth history and asked for that side's next move, and an illegal prediction at any ply forfeits the game. Errors do not corrupt subsequent positions — each prediction is independent given the true history. Autoregressive game completion has not been measured for these checkpoints and could be higher or lower; see the [game completion section of the architecture doc](https://github.com/thomas-schweich/PAWN/blob/main/docs/ARCHITECTURE.md#game-completion-rate) for the full definition. Game completion rate is a much stricter metric than per-move legal rate, and is the main signal that separates capacity between sizes.
+
+| Compound-legality detail | Value |
+|--------------------------|-------|
+| Average plies completed per game | 349 |
+| Average % of game completed | 99.88% |
+| Median forfeit ply (when forfeit) | 153 |
 
-### Accuracy Ratios
+### Accuracy ceiling
 
-PAWN is trained on uniformly random chess games, so top-1 accuracy has a hard theoretical ceiling. Ratios above 100% on the unconditioned ceiling indicate the model exploits the outcome token to make non-uniform predictions. The MC conditioned ceiling is an estimate reported as a bracket \[corrected, naive\]; see [Accuracy Ceiling Analysis](https://github.com/thomas-schweich/PAWN/blob/main/docs/ACCURACY_CEILING.md) for methodology.
+PAWN is trained on uniformly random chess games. At each position with N legal moves, the next move is drawn uniformly, so the Bayes-optimal predictor that does not know the game's outcome can do no better than 1/N at that position. Averaged over the position distribution induced by random games of up to 512 plies, the top-1 ceiling is **E\[1/N_legal\] ≈ 8.43%** (95% CI \[8.41%, 8.45%\], computed over 50,000 fresh random games — see [docs/ACCURACY_CEILING.md](https://github.com/thomas-schweich/PAWN/blob/main/docs/ACCURACY_CEILING.md)).
 
-| Ceiling | Ratio |
-|---------|-------|
-| Unconditioned (E\[1/N_legal\] = 6.52%) | 106% |
-| Bayes-optimal conditioned (MC, 128 rollouts = \[6.67, 7.34\]%) | 95–104% |
+This model's top-1 accuracy of **8.63%** is **102% of that ceiling** — i.e., essentially at the limit of what any predictor can achieve on this task without outcome information.
 
 
 ## Probe Results
 
-Linear probes trained on frozen hidden states measure how well the model's internal representations encode board-level features.
+Linear probes trained on frozen hidden states measure how well the model's internal representations encode board-level features. The model is never explicitly told about pieces, sides, or rules — these representations emerge purely from next-token prediction on random games.
 
 | Probe | Accuracy | Description |
 |-------|----------|-------------|
@@ -144,9 +155,9 @@ Edge-case diagnostics measure the model's legal move rate in specific tactical s
 | Attention heads | 8 |
 | Head dimension | 80 |
 | d_ff | 2560 |
-| Parameters | ~68.4M |
-| Vocabulary | 4,284 tokens |
-| Context length | 256 tokens |
+| Parameters | ~66.9M |
+| Vocabulary | 1,980 tokens (1,968 searchless_chess actions + 1 PAD + 11 outcome tokens) |
+| Context length | 512 tokens |
 | Normalization | Pre-norm RMSNorm |
 | FFN | SwiGLU (4x expansion) |
 | Positional encoding | Rotary (RoPE, base 10000) |
@@ -159,13 +170,14 @@ Edge-case diagnostics measure the model's legal move rate in specific tactical s
 |-----------|-------|
 | Training data | On-the-fly uniformly random legal games (no external dataset) |
 | Objective | Next-token cross-entropy (non-padding positions only) |
-| Total steps | 100,000 |
+| Outcome conditioning | Disabled (prepend_outcome=False) — pure moves, no outcome leakage |
+| Total steps | 200,000 |
 | Batch size | 256 |
-| Games seen | 25,600,000 |
-| Learning rate | 3e-4 (cosine decay with 1,000-step warmup) |
+| Total training sequences | 51,200,000 (= total steps × batch size; the published checkpoint is the best 5K-cadence step by val loss, at step 195,000 ≈ 49,920,000 sequences) |
+| Max ply per example | 512 |
+| Learning rate | 0.0003 (cosine decay with 10,000-step warmup) |
 | Optimizer | AdamW (weight decay 0.01) |
 | Precision | Mixed (AMP) |
-| Hardware | NVIDIA H200 |
 
 ## Usage
 
@@ -199,7 +211,7 @@ model.load_state_dict(weights)
 ### Finetuning with an adapter
 
 ```bash
-uv run python scripts/train_bottleneck.py \
+uv run python scripts/train.py --run-type adapter --strategy bottleneck \
     --checkpoint thomas-schweich/pawn-large \
     --pgn thomas-schweich/pawn-lichess-full \
     --bottleneck-dim 32 --lr 1e-4 --local-checkpoints
@@ -223,7 +235,7 @@ PAWN builds on ideas and tools from the following projects and publications:
 | FiLM | [Perez et al., "FiLM: Visual Reasoning with a General Conditioning Layer", AAAI 2018](https://arxiv.org/abs/1709.07871) |
 | RoSA | [Nikdan et al., "RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation", 2024](https://arxiv.org/abs/2401.04679) |
 | Linear probes | [Alain & Bengio, "Understanding Intermediate Layers Using Linear Classifier Probes", ICLR Workshop 2017](https://arxiv.org/abs/1610.01644) |
-| Intrinsic dimensionality | [Aghajanyan et al., "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning", ACL 2021](https://arxiv.org/abs/2012.13255) |
+| Searchless Chess (action vocab) | [Ruoss et al., "Amortized Planning with Large-Scale Transformers: A Case Study on Chess", 2024](https://arxiv.org/abs/2402.04494) |
 | MAIA | [McIlroy-Young et al., "Aligning Superhuman AI with Human Behavior: Chess as a Model System", KDD 2020](https://arxiv.org/abs/2006.01855) |
 | AlphaZero | [Silver et al., "A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go through Self-Play", Science 2018](https://arxiv.org/abs/1712.01815) |
 | Leela Chess Zero | [github.com/LeelaChessZero/lc0](https://github.com/LeelaChessZero/lc0) |