OWT Small Reproduction, Context 256
This repository contains the completed tokenizer and generator checkpoints for
the owtsmall-repro-ctx256 run.
Artifacts
vqvae/checkpoint-iter-25788.pt: 10-epoch multiscale VQ-VAE tokenizer.ncp/checkpoint-iter-17192.pt: 5-epoch next-concept generator with one output head per predicted scale.config/: the exact training configurations.evaluation/: aggregate generation-perplexity results over five seeds.
Both checkpoints are native PyTorch training checkpoints. Each contains the model state, optimizer and scheduler state, saved configuration, training step, best validation loss, scaler state, and per-rank RNG state.
Architecture
The tokenizer uses 12 multiscale levels at token counts
[1, 4, 16, 25, 36, 64, 81, 121, 144, 169, 225, 256], with a codebook size of
16,384 at every level. It has 196.82M parameters and was trained for 10 epochs
on small_owt GPT-2 token IDs with document-aware, document-length-weighted
sampling.
The generator has 12 transformer layers, 12 attention heads, embedding width
768, context length 256, and 11 separate Linear(768, 16384) output heads. It
has 228.82M parameters and was trained for 5 epochs using the frozen tokenizer.
Both stages used torch.compile.
Results
| Metric | Result |
|---|---|
| Tokenizer validation reconstruction loss | 0.003189 |
| Tokenizer validation reconstruction accuracy | 99.93% |
| Generator validation loss | 4.579768 |
| Generator validation accuracy | 45.3499% |
| Generator last-level loss | 0.62108 |
| Generator last-level accuracy | 82.7406% |
| Random generation PPL | 155.3968 ± 1.116 SE |
| Random generation entropy | 4.4579 nats |
top_k=50, top_p=0.95 generation PPL |
50.8859 ± 0.407 SE |
top_k=50, top_p=0.95 generation entropy |
3.7862 nats |
Generation PPL uses GPT-2 Large as the reference model, 128 samples per seed, five seeds, temperature 1.0, and one provided coarse level. Validation examples are sampled with document offsets weighted by document length.
Shared-head follow-up
ncp-sharedhead/checkpoint-iter-17192.pt is the matched 5-epoch follow-up in
which all 11 predicted scales reuse one strict Linear(768, 16384) output head.
It has 102.83M parameters; all other generator and training settings are kept
the same, with batch size 96 and gradient accumulation 4 preserving the
effective batch of 384 per rank.
| Metric | Shared head | Separate heads |
|---|---|---|
| Validation loss | 5.014824 | 4.579768 |
| Validation accuracy | 40.9332% | 45.3499% |
| Last-level loss | 0.780699 | 0.621080 |
| Last-level accuracy | 79.9076% | 82.7406% |
| Random generation PPL | 180.2808 ± 2.4756 SE | 155.3968 ± 1.1160 SE |
| Random generation entropy | 4.4786 nats | 4.4579 nats |
top_k=50, top_p=0.95 PPL |
55.7854 ± 0.9089 SE | 50.8859 ± 0.4070 SE |
top_k=50, top_p=0.95 entropy |
3.7805 nats | 3.7862 nats |
The exact shared-head configuration is in config/ncp-sharedhead.yaml, with
aggregate evaluation outputs under evaluation/sharedhead/.
Semantic substitution two-stage control (10k per component)
substitution-two-stage-10k-decay34384-v1 contains the completed independent tokenizer 10,000-update stage and fresh-generator 10,000-update stage, both using a 34,384-update LR decay horizon and frozen pinned RoBERTa input features. Full checkpoints and inference exports at the evaluated generator steps 5k/10k, all evaluation samples/results, exact source/configuration records and checksums are included. Final generation PPL: random 661.58 ± 25.58 SE; top-k50/top-p0.95 76.25 ± 2.14 SE. These results use a different training budget and tokenizer input architecture from the original baseline above.
Encoder substitution: generator extended to 34,384 updates
substitution-two-stage-tokenizer10k-generator34384-v1 preserves the completed generator continuation with its independently trained 10k tokenizer kept frozen. Full checkpoints, inference exports, all evaluation samples/results, source and continuation records are included for generator updates 17,192, 25,000 and 34,384. Final generation PPL: random 516.08 ± 20.86 SE; top-k50/top-p0.95 75.22 ± 1.96 SE. The lowest measured truncated PPL was 72.36 ± 1.68 SE at 17,192.