1.5TB reduced to 214GB: Tencent releases extreme quantized version of Hy4 preview

Beating AI News: Just after the Hy4 preview went open-source, Tencent has released an extreme quantized version of its Hunyuan model. The original model weights are nearly 1.5TB, while the new GGUF version is only around 214GB, drastically lowering the local deployment barrier for this 770B MoE model. Tencent did not uniformly quantize the entire model to 1.25-bit; instead, it applied different quantization levels based on each layer’s sensitivity to precision: non-critical layers are compressed to as low as ~1.31-bit, while sensitive layers retain 2-bit or higher precision, resulting in an average of ~2.38 bits per weight (bpw). In four benchmarks provided by Tencent, the quantized version only dropped 0.2 to 1.6 points compared to the BF16 original. After compression, Tencent also tested heterogeneous device joint inference with prima.cpp. A setup consisting of an RTX 4090 laptop and a 4-A4000 server, with only 80GB of total VRAM and 64GB of RAM, achieved an inference speed of 1.02 tokens per second—roughly 6 times faster than running the model offloaded on the laptop alone. Multiple devices with different configurations can also jointly share the model inference workload.