---
license: mit
---

nanochat-d34 model. It was pretrained like this:

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=34 --device_batch_size=4 --target_param_data_ratio=40 --save_every=5000 --run=d34
```

On an 8XH100 node, which ran for \~100 hours (\~4 days) and cost \~$2,500. Notably, note that it is 2X longtrained compared to Chinchilla, i.e. the param:token ratio is overridden from 20 up to 40. This means the model was trained longer that is compute optimal, just to squeeze a bit more capability into a bit smaller package.

Some of the notable stats of the model are as follows:

```
- depth: 34
- max_seq_len: 2048
- target_param_data_ratio: 40
- device_batch_size: 4
- total_batch_size: 524,288
- Number of parameters: 2,217,082,880
- Number of FLOPs per token: 1.426509e+10
- Calculated number of iterations: 169,150
- Number of training tokens: 88,683,315,200
- Tokens : Params ratio: 40.0000
- DDP world size: 8
- Minimum validation bpb: 0.7045
- Final validation bpb: 0.7045
- MFU %: 47.40%
- Total training flops: 1.265075e+21
- Peak memory usage: 69811.85MiB
```

The achieved **CORE score** of this base model is **0.3382**. (For comparison, d32 is 0.3168, d20 "speedrun" is 0.22, GPT-2 is 0.25).

This upload allows you to skip base model pretraining and focus on finetuning, which is a lot cheaper, and saves you $2,500 of cost. To incorporate it into your nanochat repo it's a bit janky right now, basically download these files and then:

- the `token_bytes.pt`, `tokenizer.pkl` have to go into ~/.cache/nanochat/tokenizer directory
- the `meta_169150.json` and `model_169150.pt` have to go into ~/.cache/nanochat/chatsft_checkpoints/d34/
- 
I'll figure out how to make this less janky in the future, and to make nanochat play nicer with huggingface infra.