This quant needs fixing
#1
by trohrbaugh - opened
config.json's ignore is missing three entries present in the model card's creation snippet, and 86 weight_scale tensors exist for modules the recipe excludes, which trips assert key == "w2_weight" in vLLM's sink_experts loader.
Solution 2: Edit config.json (Permanent Fix)
- Locate the "ignore" array inside "quantization_config".
- Change "re:.gate." to a stricter string so it only targets your model's actual unquantized text/vision gates, leaving the MoE layers alone. Change it to:
json
"re:.*\.gate$"