Large language models (LLMs) have remarkable capabilities, but their size presents a major challenge. A model with billions of parameters can require tens or even hundreds of gigabytes of expensive GPU memory (VRAM) just to load. This is where LLM quantization comes in.
LLM quantization is a compression technique that reduces the memory footprint of a model by storing its numbers with less precision. Instead of using a 16-bit format, it might use 8-bit or 4-bit integers. This allows you to run larger, more capable models on smaller, more accessible hardware.
Key takeaways
- Quantization reduces an LLM's size by storing its parameters (weights) with fewer bits, such as 4-bit integers instead of 16-bit floating-point numbers.
- The primary benefits are fitting larger models into available VRAM and, with the right setup, increasing inference speed.
- Advanced methods like GPTQ and AWQ intelligently quantize models to preserve accuracy, but some performance loss is always possible.
- Accuracy degradation depends on the model, quantization method, task, and the data used for calibration. It is not a fixed percentage (Source name).
- Real-world memory savings are less than the ideal ratio (e.g., 4x for 16-bit to 4-bit) due to overhead from scaling factors, metadata, and unquantized parts of the model like the KV cache.
What Is LLM Quantization and Why Does It Matter?
LLM quantization reduces a model's memory footprint by converting its weights from high-precision numbers to a lower-precision format. This compression makes it possible to run larger models on hardware with less memory, like consumer GPUs. The goal is to shrink the model while having the smallest possible impact on its predictive accuracy.
Models are typically trained using 32-bit (FP32) or 16-bit (FP16/BF16) floating-point numbers. A floating-point number can represent a wide range of values, including decimals, which is essential for learning the subtle patterns in data during how models are trained.
However, this precision comes at a cost. A 7-billion parameter model in FP16 needs about 14 GB of VRAM for its weights alone. Quantization converts these FP16 numbers into smaller formats like 8-bit integers (INT8) or 4-bit integers (INT4). An integer is a whole number. This conversion reduces memory requirements and can also speed up calculations if the hardware has specialized support for integer math.
How Does LLM Quantization Work, Step by Step?
The process maps a range of high-precision floating-point numbers to a smaller set of low-precision integers. For each group of weights, it calculates a scaling factor to convert them into integers. During inference, these integers are often multiplied by the scale to be de-quantized back into an approximate float before computation, saving memory storage and bandwidth.
Here is a simplified, step-by-step look at the mechanism.
Step 1: Choose a Granularity
The algorithm first decides how to group the numbers. It can apply one scaling factor to an entire tensor (a multi-dimensional array of weights), to each row or column (per-channel), or to smaller blocks of numbers (e.g., groups of 64 or 128 weights). Smaller groups preserve more local precision but add slightly more overhead for storing the extra scaling factors.
Step 2: Find the Scaling Factor
For a given group of floating-point weights, the algorithm finds the value with the largest magnitude (the "absolute maximum"). This abs_max value determines the range that needs to be compressed. The scaling factor is calculated to map this range into the target integer range.
For example, a signed 4-bit integer (INT4) can represent whole numbers from -8 to 7. To map our floats to this range, we can use a symmetric quantization scheme where the scale is scale = abs_max / 7.
Step 3: Map Floats to Integers
Each floating-point weight in the group is divided by the scaling factor and then rounded to the nearest whole number. This is the core conversion step.
quantized_value = round(original_value / scale)
The original high-precision floating-point number is now represented by a low-precision integer.
Step 4: De-quantize for Computation
To perform a calculation like matrix multiplication, the model often needs to convert the integers back to floating-point numbers. This is called de-quantization.
approximated_value = quantized_value * scale
This "approximated_value" will be very close, but not identical, to the original weight. The small difference is the quantization error. Some modern hardware and software can perform mathematical operations directly on the quantized integers, avoiding this step and offering greater speedups.
Step 5: Handle Outliers
Some weights are far more important than others. These "outliers" can have a large impact on the model's output. Naively quantizing them can significantly harm performance. Sophisticated methods have ways to handle this. For instance, the LLM.int8() method uses a mixed-precision approach, processing most values in INT8 but identifying and handling outliers in full FP16 precision (Source name).
What's the Difference: Weight vs. Activation Quantization?
Weight-only quantization compresses only the model's static parameters, reducing storage and memory bandwidth. Weight-and-activation quantization also compresses the dynamic, intermediate results (activations) that flow between layers. While the latter can offer greater speedups on compatible hardware, it is more complex because activation values change with every input.
Weight-Only Quantization
This is the most common approach for LLMs, used by methods like GPTQ and AWQ.
- What it is: Only the model's weights, which are fixed after training, are converted to a low-bit format like INT4.
- Benefit: Its primary advantage is reducing the model's memory footprint and the bandwidth needed to read weights from memory. This is especially helpful for large models where memory is the main bottleneck.
- Challenge: The activations—the numbers that represent the data flowing between layers—remain in high precision (e.g., FP16). Before each computation, the weights must be de-quantized back to FP16. This de-quantization step adds a small amount of overhead.
Weight-and-Activation Quantization (e.g., W8A8)
This is a more aggressive approach that quantizes both weights and activations.
- What it is: Both the static model weights and the dynamic activations are converted to a low-bit format, often INT8.
- Benefit: This can lead to significant speedups. When both inputs to a matrix multiplication are integers, the computation can be performed using highly efficient integer arithmetic units on modern GPUs, reducing both memory traffic and computational time.
- Challenge: Activations are much harder to quantize. Their range of values can vary dramatically depending on the input prompt. This requires careful calibration or dynamic quantization schemes to avoid large accuracy losses (Source name).
How Do You Choose a Quantization Method? (GPTQ vs. AWQ vs. bitsandbytes)
Choose a method based on your goal and hardware. For local experimentation or memory-efficient fine-tuning on consumer GPUs, bitsandbytes is a common choice. For high-throughput production serving on NVIDIA GPUs, AWQ or GPTQ are supported by frameworks like vLLM.
The right method depends on your priorities: ease of use, inference speed, memory savings, or fine-tuning capability.
| Method / Format | Best For | Key Feature | Accuracy | Ease of Use |
|---|---|---|---|---|
bitsandbytes |
Local experimentation, academic research, and accessible fine-tuning. | Offers 4-bit and 8-bit quantization in PyTorch. Enables memory-efficient fine-tuning methods like QLoRA, which uses a frozen 4-bit base model (Source name). | Good, but can be slightly lower than accuracy-focused methods like GPTQ/AWQ. | Very High |
| GPTQ | Production inference on GPUs where accuracy is critical. | Aims to minimize quantization reconstruction error using a calibration dataset and approximate second-order information (Source name). | Very Good | Medium |
| AWQ | High-throughput production inference on GPUs where speed is paramount. | Uses activation statistics to identify and protect salient weights, allowing for fast custom kernels (Source name). | Very Good | Medium |
A simple decision process:
- Are you fine-tuning? Use QLoRA with
bitsandbytes. - Are you deploying to a production server with NVIDIA GPUs? Evaluate
AWQandGPTQfor the best speed/accuracy trade-off for your model. Serving frameworks like vLLM support both formats, and performance depends on hardware-specific kernels (Source name).
A Worked Example: Quantizing a 7B Model
To understand the practical impact of quantization, let's calculate the memory requirements for a 7-billion-parameter model. This example highlights why real-world memory savings don't always match the simple theoretical ratio.
Baseline: 16-bit Floating Point (FP16)
- Each parameter is stored as a 16-bit number, which is 2 bytes.
- Weight Memory: 7,000,000,000 parameters × 2 bytes/parameter = 14,000,000,000 bytes.
- Total: 14 GB just for the model weights.
Ideal Quantization: 4-bit Integer (INT4)
- Each parameter is stored as a 4-bit number, which is 0.5 bytes.
- Weight Memory: 7,000,000,000 parameters × 0.5 bytes/parameter = 3,500,000,000 bytes.
- Total: 3.5 GB for the model weights.
This represents a theoretical 4x reduction in memory for the weights, or a 75% savings (Source name). However, the total VRAM used will be higher.
Real-World Memory Usage
The 3.5 GB figure is an absolute minimum. Actual memory usage is higher due to several factors:
- Scales and Metadata: Each group of quantized weights needs its own scaling factor. These scales are usually stored in FP16, adding a few hundred megabytes of overhead.
- Unquantized Layers: For accuracy, it's common practice to leave some parts of the model in full precision. The initial embedding layer and the final prediction layer are often kept in FP16, which can add hundreds of megabytes.
- The KV Cache: This is a major consumer of memory that is not affected by weight quantization. The KV cache stores intermediate attention values to speed up token generation. Its size depends on the context length and batch size, and it can easily consume several gigabytes of VRAM for long sequences.
- Framework Overhead: The inference engine and CUDA kernels also require their own memory buffers, further increasing the total VRAM footprint.
In practice, a 4-bit quantized 7B model might require 5-6 GB of VRAM to start, with more needed as the context window fills. While this is a significant increase from the "ideal" 3.5 GB, it is still a massive saving compared to the 14 GB required for the FP16 version.
What to learn next
- Synthetic Data LLM Training: A Guide to Avoiding Model Collapse - Learn the risks and rewards of synthetic data LLM training. This guide covers how to avoid model collapse, filter data, and mix synthetic an
- The KV Cache LLM Explained: A Complete Technical Guide: Learn about the other major factor influencing LLM memory usage and inference speed.
- LoRA Fine-Tuning: A Complete Guide to Low-Rank Adaptation: Discover how quantization enables memory-efficient fine-tuning with methods like QLoRA.
- Fine Tuning vs RAG: Which Method Should You Choose?: Understand the different ways to adapt LLMs for your specific tasks and data.
Sources and further reading
- On the Practical Usability of 4-bit Quantization for Large Language Models
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- QLoRA: Efficient Finetuning of Quantized LLMs
- vLLM Documentation: Quantization
- vLLM Documentation: BitsAndBytes Quantization
