mchlxo/pythia-gsm8k-70m

QLoRA adapter for EleutherAI/pythia-70m (70M parameters) fine-tuned on the GSM8K training set.

Training details

Setting	Value
Method	QLoRA (4-bit NF4, double quant)
LoRA rank	16
LoRA alpha	32
Target modules	all linear layers
Epochs	3
Learning rate	2e-4
Scheduler	cosine
Optimizer	paged_adamw_8bit
Gradient checkpointing	no

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-70m")
model = PeftModel.from_pretrained(base, "mchlxo/pythia-gsm8k-70m")
tokenizer = AutoTokenizer.from_pretrained("mchlxo/pythia-gsm8k-70m")

Purpose

This adapter was trained as part of a scaling study on benchmark contamination. The study measures the accuracy gap between the GSM8K test set (potentially contaminated by fine-tuning) and Apple GSM-Symbolic (finetuning-proof), across the full Pythia model family.