feat: add importance-aware mixed-precision quantization

[wangwenmingaa]

This PR introduces and implements an importance-aware mixed-precision quantization method for large language models, with the following key features:

Importance-Guided Quantization Strategy:
We evaluate the contribution of each layer (or module) to the overall model performance and dynamically assign quantization precision accordingly. Empirical analysis shows that the first few and last few layers are typically more critical, so we keep them in full precision (e.g., FP16/FP32), while less sensitive middle layers are aggressively quantized.

Mixed-Precision Configuration:
High-importance layers: No quantization (kept in FP16/FP32)
Medium-importance layers: Quantized to FP8
Low-importance layers: Quantized to FP4
Strict Average Bitwidth Constraint (≤ 8-bit):
The overall average bitwidth across all layers is constrained to no more than 8 bits, ensuring efficient resource usage while maximizing performance.

Outperforms FP8 per-channel Baseline:
Accuracy: Achieves an average +1.04% accuracy gain over FP8 per-channel quantization on challenging benchmarks including AIME 2024–2025, LiveCodeBench, and MMLU-Pro, demonstrating superior preservation of model capabilities under aggressive quantization.
Inference Speed: Runs faster than the FP8 per-channel quantized baseline under identical hardware conditions, thanks to reduced memory bandwidth pressure from mixed-precision weight layouts.
Practical & Plug-and-Play: Requires no architectural modifications—only a principled, importance-aware assignment of quantization policies—making it immediately deployable for real-world large language model inference.

[gemini-code-assist]

Summary of Changes

Hello @wangwenmingaa, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a novel importance-aware mixed-precision quantization method designed to optimize large language models. By intelligently assigning different precision levels (FP4, FP8, FP16/FP32) to layers based on their impact on model performance, it aims to significantly reduce model size and improve inference speed while maintaining or even enhancing accuracy compared to standard FP8 quantization. The approach is presented as a practical, plug-and-play solution for LLM deployment.

Highlights

• Importance-Guided Quantization: Implements a strategy to dynamically assign quantization precision based on each layer's contribution to model performance, keeping critical layers in full precision and aggressively quantizing less sensitive ones.
• Mixed-Precision Configuration: Utilizes a tiered approach: high-importance layers remain FP16/FP32, medium-importance layers are quantized to FP8, and low-importance layers to FP4.
• Bitwidth Constraint: Ensures an overall average bitwidth across all layers does not exceed 8 bits for efficient resource usage.
• Performance Gains: Achieves an average +1.04% accuracy gain over FP8 per-channel quantization on benchmarks and runs faster due to reduced memory bandwidth pressure.
• Plug-and-Play Implementation: The method requires no architectural modifications, making it immediately deployable for real-world LLM inference.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Code Review

This pull request introduces a new example script for importance-aware mixed-precision quantization. The script is well-structured and serves as a good demonstration. I've provided a couple of suggestions to improve the readability and maintainability of the layer configurations within the script, mainly by clarifying comments and sorting layer indices.

[wangwenmingaa]

@dsikka @kylesayrs This PR is ready for review. Could someone please take a look? Thanks!

[mratsim]

I think if we give such an example it should be grounded on academic research so that people 5 years from now have an easier time to challenge the current status quo.

For example when doing importance aware quantization to quantize GLM-4.6 to fit in 192GB VRAM, this is what I mention: https://huggingface.co/mratsim/GLM-4.6-EXL3

Quantization theory and heuristics for manual tuning

Layers to quantize

Quantization should be focused on Linear layers (also called Dense or Fully-Connected layers i.e. MatMul+Bias)
In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1]

LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the
LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression.
Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized.

This is also reported in Intel and Nvidia repo:

• Why is LayerNorm not quantized to int8 in PTQ? intel/neural-compressor#1963 (comment)
• Q-DQ nodes for int8 LayerNorm NVIDIA/TensorRT#4084 (comment)

EXL3 can only quantize linear layers.

Tensors to up-quantize

If there is enough bits, down projections should be prioritized.

According to [4]

Fig. 3: Maximum absolute value over layers for a LLaMA3-8B.
Each color represent a different projection and we clearly see that down_proj has the biggest
spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model

According to [5]

Figure 5(a) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting
that weight outliers are concentrated in the down-projection matrices Wdown
ℓ of the second layer and
the last two layers. Figures 5(b) and 5(c) provide detailed visualizations of these outliers in the last
two layers.

Mixture-of-Experts quantization (MoE)

Mixture-of-Experts require specific quantization techniques.

Mixed-precision quantization

Some layers have a higher impact on LLM performance.
According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers.
According to [3] on 2-bit quantization:

• quantizing expert FFN layers do not seriously impact model quality
• quantizing cross-attention has some impact
• quantizing self-attention has a large impact
• quantizing dense FFN has a very significant impact

Hence to preserve model quality we should choose not to quantize dense FFN layers and self-attention layers.

We notice that:

• official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16:
  • https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json
• NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16:
  • https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json

Layers with high-impact

According to [2], giving more bits to the first k blocks have a significantly higher impact on model quality than for the same last k blocks.

Expert quantization

When quantizing MoE, quantizing activations is tricky as only a subset of experts are activated per request.

EXL3 has the tooling in-place to ensure all experts are activated during quantization, though it is unsure if the dataset should be expanded to be diverse enough so that all experts have a high likelyhood of taking the full range of values they can exhibit to avoid clipping.

References

1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)
   Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia
   https://arxiv.org/pdf/2506.12044

2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)
   Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen
   https://arxiv.org/pdf/2406.08155v1

3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)
   Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla
   https://arxiv.org/pdf/2310.02410

4. Precision Where It Matters: A Novel Spike
   Aware Mixed-Precision Quantization Strategy for
   LLaMA-based Language Models (2025)
   Lucas Maisonnave, Cyril Moineau, Olivier Bichler, and Fabrice Rastello
   https://arxiv.org/pdf/2504.21553

5. Systematic Outliers in Large Language Models (2025)
   Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang
   https://arxiv.org/pdf/2502.06415v2