GeoMaciolek/Qwen3.8-32B-Jumbo-Q4_K_S-GGUF
Text Generation • 31B • Updated • 463 • 4
Geo Maciolek PRO
GeoMaciolek
AI & ML interests
None yet
Recent Activity
reacted to dipankarsarkar's post with 🔥 about 24 hours ago
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: https://huggingface.co/papers/2609.30074
Dataset: https://huggingface.co/datasets/dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls? new activity 4 days ago
Gryphe/Pantheon-Reasoning-27B:Any chance for a 1.1 on updated Qwen 3.8? liked a model 5 days ago
sky7350/Mica-v0.1-4BOrganizations
None yet
datasets 0
None public yet