Chess-SFT-Models

SFT (thinking) chess models from the compute-allocation study. One subfolder per model, named model_{size}_{pretraining_tokens} (tokens in billions, 2 s.f.). Each subfolder has its own model card with a pass@1 / pass@16 figure (score vs pretraining tokens, pretraining base vs SFT-thinking).

Loading

Models ship a custom tokenizer (tokenizer.py), so trust_remote_code=True is required. The remote-code loader ignores subfolder=, so snapshot the model folder locally first and load from that path:

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "model_50m_17B"   # see the table below
path = snapshot_download("pavelslab-nyu/Chess-SFT-Models", allow_patterns=f"{name}/*") + f"/{name}"
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Generating a rollout

generate.py at the repo root ships a default prompt (a test_B0 position):

python generate.py --model model_50m_17B          # uses the default prompt
python generate.py --model model_680m_6.4B --prompt "1. d4 Nf6 2. c4 g6 <T>"

Note: append the <T> (think) token to the end of your prompt — it's the cue for the model to start its rollout. A prompt without a trailing <T> will not generate as intended.

Models & evaluation

pass@k on the multi-turn chess benchmark (test_B0), SFT-thinking model.

model	size	pretrain tokens	pass@1	pass@16
model_20m_0.53B	20m	0.53B	11.9%	71.6%
model_20m_1.6B	20m	1.6B	20.9%	87.6%
model_20m_2.6B	20m	2.6B	18.7%	84.4%
model_20m_5.3B	20m	5.3B	28.2%	91.7%
model_20m_11B	20m	11B	28.0%	88.5%
model_20m_16B	20m	16B	36.8%	93.2%
model_20m_21B	20m	21B	37.8%	93.2%
model_20m_32B	20m	32B	35.1%	92.7%
model_20m_40B	20m	40B	36.9%	95.5%
model_20m_53B	20m	53B	33.2%	94.2%
model_32m_6.9B	32m	6.9B	—	—
model_32m_14B	32m	14B	—	—
model_50m_0.23B	50m	0.23B	3.3%	26.7%
model_50m_0.69B	50m	0.69B	27.0%	91.9%
model_50m_1.1B	50m	1.1B	36.1%	95.7%
model_50m_2.3B	50m	2.3B	44.6%	96.6%
model_50m_4.6B	50m	4.6B	55.9%	98.5%
model_50m_9.2B	50m	9.2B	57.2%	98.7%
model_50m_17B	50m	17B	62.5%	98.9%
model_50m_23B	50m	23B	58.8%	98.9%
model_50m_41B	50m	41B	60.0%	98.7%
model_100m_0.53B	100m	0.53B	31.5%	92.3%
model_100m_2.1B	100m	2.1B	59.4%	98.5%
model_100m_4.3B	100m	4.3B	62.7%	98.9%
model_100m_8.0B	100m	8.0B	64.6%	100.0%
model_200m_0.27B	200m	0.27B	9.2%	63.0%
model_200m_0.53B	200m	0.53B	52.2%	98.1%
model_200m_1.1B	200m	1.1B	54.6%	98.1%
model_200m_2.1B	200m	2.1B	66.7%	98.9%
model_200m_4.0B	200m	4.0B	72.3%	99.6%
model_200m_5.3B	200m	5.3B	73.3%	99.6%
model_200m_11B	200m	11B	73.6%	99.8%
model_200m_21B	200m	21B	76.7%	99.8%
model_200m_40B	200m	40B	74.7%	99.1%
model_200m_53B	200m	53B	77.2%	99.6%
model_410m_2.0B	410m	2.0B	—	—
model_410m_2.6B	410m	2.6B	—	—
model_680m_0.080B	680m	0.080B	2.5%	27.8%
model_680m_0.32B	680m	0.32B	26.7%	88.0%
model_680m_0.64B	680m	0.64B	54.5%	97.6%
model_680m_1.2B	680m	1.2B	68.6%	99.4%
model_680m_1.6B	680m	1.6B	71.5%	99.4%
model_680m_3.2B	680m	3.2B	77.3%	99.8%
model_680m_6.4B	680m	6.4B	80.5%	99.8%
model_680m_12B	680m	12B	80.5%	99.8%
model_680m_16B	680m	16B	80.7%	100.0%
model_680m_24B	680m	24B	84.0%	99.8%
model_680m_32B	680m	32B	85.3%	100.0%
model_680m_48B	680m	48B	81.1%	100.0%