Vega-1.6-128M-Base

Note: AI was used in the creation of this project. But that's why you're here, isn't it?

---

This is the base model for Vega-1.6-128M, an experimental SLM trained on various datasets. (Ha! I've learned from last time!)

First corpus was 3B toks of 33.3/33.3/33.3 fineweb, fineweb-edu, and cosmopedia. Second corpus was 5B tokens taken from a 35/30/20/15 split of NPset-2-Python-Edu, open-web-math, cosmopedia, and fineweb-edu.

Benchmarks

1019 elo on Bananamind Base Bench 1.1, putting it in 17th place as of writing, ahead of GPT-X-125M and behind Supra2-100M.

• Hellaswag: 29.62%
• ARC-E: 39.64%
• ARC-C: 23.89%
• PIQA: 59.08%
• Arithmark-3: 35.60%

Model specs

• Layers: 16
• Hidden size: 768
• Attention heads: 12
• KV heads: 6
• Context length: Trained on 2,048 tokens
• Intermediate size: 2048
• Vocabulary: 32,000
• Parameters: 128,410,368