Step 6: Test Quantization (Compare Performance and Memory)

Section 6 of 8-~ 11 min read-Synced from Cuantum content

Quantization is one of the most powerful levers you have for reducing deployment costs and improving inference speed. But it's not free—you're trading precision for efficiency, and the impact on model quality varies depending on the quantization method, the model architecture, and your specific task. This step walks you through a systematic comparison so you can make an informed decision based on data, not guesswork.

The goal here is to run a controlled experiment comparing two configurations:

Non-quantized serving: Your baseline. This is the full-precision (or bfloat16) model you've been testing so far.
Quantized serving: The same model, but with weights compressed to lower precision—typically 4-bit or 8-bit.

You'll measure three things: latency, memory usage, and quality. The first two come from your monitoring script. The third requires manual inspection or automated evaluation, depending on how rigorous you want to be. For most real-world deployments, a combination of automated metrics and human review works best.

Option A: Quantize the Base Model for Serving

The most production-relevant approach is to serve a quantized checkpoint directly through vLLM. This gives you realistic measurements of how quantization affects inference in your actual serving environment, not just in a toy script. vLLM supports several quantization formats, including AWQ (Activation-aware Weight Quantization) and GPTQ (Generative Pre-trained Transformer Quantization). Both are post-training quantization methods that compress model weights without requiring you to retrain or fine-tune.

AWQ is optimized for preserving activation magnitudes, which tends to result in better quality for instruction-following models. GPTQ is more widely supported and has been around longer, so you'll find more pre-quantized checkpoints on Hugging Face. The performance difference between them is usually small—typically within 1-2% on most benchmarks—but AWQ often edges ahead for conversational and instruction-tuned models.

The workflow for testing quantized serving looks like this:

Find or create a quantized checkpoint. The easiest path is to search Hugging Face for a pre-quantized version of your base model. For example, if you're using Mistral-7B-Instruct-v0.2, search for "Mistral-7B AWQ" or "Mistral-7B GPTQ". Community members and organizations like TheBloke maintain extensive collections of quantized models. If you can't find a pre-quantized version, you can create one yourself using the auto-gptq or autoawq libraries, but that's beyond the scope of this chapter.
Serve the quantized checkpoint with vLLM. The command is nearly identical to what you used before. For AWQ models, vLLM detects the quantization automatically from the model config. For GPTQ, you may need to pass --quantization gptq explicitly. Check the vLLM documentation for your specific version, as flag names occasionally change.
Re-run your monitoring script. Use the exact same prompts you tested earlier. This is critical—if you change the prompts, you're introducing a variable that makes comparison impossible. Your monitor_requests.py script already logs everything you need: latency, response length, and GPU memory. Run it against the quantized server and save the output to a separate JSONL file so you can compare side-by-side.

Here's what a typical vLLM command for serving a quantized model might look like:

vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ \  --enable-lora \  --lora-modules mylora=/path/to/your/lora/adapter \  --max-model-len 4096 \  --gpu-memory-utilization 0.85

Notice that you can still load LoRA adapters on top of a quantized base model. vLLM applies the adapter in full precision during inference, so you don't lose the benefits of fine-tuning. The memory overhead of the adapter is negligible compared to the base model, so the total memory savings from quantization remain significant.

After running your monitoring script against both the quantized and non-quantized servers, you'll have two JSONL files. Load them into a spreadsheet or pandas DataFrame and calculate summary statistics: average latency, p95 latency, average GPU memory usage, and memory variance. The comparison should look something like this:

Configuration       Avg Latency (s)  P95 Latency (s)  Avg GPU Mem (MB)Non-quantized       2.15             2.68             14,230AWQ 4-bit           1.78             2.21             9,120

In this hypothetical example, quantization reduces latency by about 17% and cuts memory usage by 36%. Those are real savings—enough to fit the model on a smaller GPU, or to increase your batch size and handle more concurrent requests. But the numbers you see will depend on your hardware, your model, and your specific inference patterns.

The memory savings are usually the more dramatic benefit. A 7B parameter model in bfloat16 uses roughly 14GB of VRAM. The same model quantized to 4-bit uses around 3.5GB for weights alone (plus overhead for activations and KV cache). This means you can serve a 7B model on a consumer GPU like an RTX 4090, or run multiple replicas on a single A100. For production deployments, this translates directly to cost: fewer GPUs, lower cloud bills, and more headroom for traffic spikes.

Latency improvements are less predictable. Quantized models have smaller memory footprints, which can reduce memory bandwidth bottlenecks, but the quantized operations themselves may be slower depending on your GPU and kernel implementations. On modern GPUs with good INT4 support (like A100 or H100), you'll often see latency improvements. On older GPUs, the benefit is smaller or even negative. The only way to know for sure is to measure on your target hardware.

Option B: Measure with bitsandbytes in a Local Inference Script

If you're not ready to set up a quantized checkpoint for vLLM—maybe you want a quick feasibility check before committing to the full workflow—you can test quantization locally using the bitsandbytes library. This won't give you vLLM's batching and scheduling optimizations, so the absolute latency numbers won't match production, but it's useful for two things: understanding the memory impact and evaluating whether quantization degrades quality for your specific use case.

The bitsandbytes library integrates directly with Hugging Face Transformers, making it trivial to load a model in 4-bit or 8-bit mode. Here's a minimal example that loads Mistral-7B-Instruct in 4-bit and generates a response:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torch model_name = "mistralai/Mistral-7B-Instruct-v0.2" bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4") tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModelForCausalLM.from_pretrained(    model_name,    device_map="auto",    quantization_config=bnb_config) prompt = "Explain why quantization helps reduce inference cost."inputs = tokenizer(prompt, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=120)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Let's unpack the configuration. The BitsAndBytesConfig object controls how quantization is applied. load_in_4bit=True tells the library to quantize model weights to 4 bits. bnb_4bit_compute_dtype=torch.float16 specifies that intermediate computations should happen in float16, not int4—this preserves accuracy during the forward pass while keeping memory usage low. bnb_4bit_use_double_quant=True enables a nested quantization scheme where even the quantization constants themselves are quantized, squeezing out a bit more memory savings. And bnb_4bit_quant_type="nf4" selects the NF4 (Normal Float 4) quantization format, which is optimized for weights that follow a normal distribution—common in most LLMs.

The device_map="auto" argument tells Transformers to automatically distribute the model across available GPUs and CPU RAM if needed. For a 7B model in 4-bit, everything should fit comfortably on a single GPU, but this flag makes the code robust to different hardware configurations.

After loading the model, you can generate text just like you would with a full-precision model. The interface is identical, which is the beauty of bitsandbytes—quantization is transparent to the rest of your code. You can run this script with a few different prompts and manually inspect the outputs. Look for:

Coherence: Does the response stay on topic? Does it follow the instruction?
Fluency: Are there awkward phrasings, repetitions, or grammatical errors that weren't present in the full-precision version?
Factual accuracy: For knowledge-intensive tasks, does the model still produce correct information?

In most cases, 4-bit quantization with NF4 produces outputs that are nearly indistinguishable from the full-precision model. You might see very subtle differences—slightly less confident predictions, marginally less natural phrasing—but for the majority of real-world applications, the quality is acceptable. If you're working on a highly sensitive task where even small degradations matter (legal document analysis, medical question answering), you'll want to run a more rigorous evaluation with a held-out test set and automated metrics. But for conversational AI, content generation, and most instruction-following tasks, manual inspection of a dozen or so outputs is usually sufficient.

To measure memory usage in this local setup, you can use torch.cuda.memory_allocated() before and after loading the model:

import torchprint(f"Memory before loading: {torch.cuda.memory_allocated() / 1e9:.2f} GB")model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", quantization_config=bnb_config)print(f"Memory after loading: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

For a 7B model, you should see the full-precision version consuming around 14GB and the 4-bit version consuming around 4-5GB (the extra overhead comes from activations and intermediate tensors during the forward pass). This confirms the memory savings you'd expect from quantization theory, and gives you confidence that the same savings will translate to vLLM when you deploy the quantized checkpoint.

This local testing approach doesn't replace the full vLLM benchmark—it's not measuring request batching, concurrent load, or realistic serving latency. But it's a fast sanity check that lets you iterate on quantization settings before committing to the full deployment pipeline. If quality looks good here, you can proceed with confidence to Option A. If quality is unacceptable, you know quantization won't work for your task, and you need to explore other optimization strategies—distillation, pruning, or switching to a smaller model family.

One final note on quantization and LoRA: If you're planning to serve a quantized base model with a LoRA adapter, make sure the adapter was trained in a compatible way. If you trained your LoRA on a full-precision model, it will still work when applied to a quantized base, but the quality might degrade slightly because the adapter's learned updates assume full-precision activations. For maximum quality, consider using QLoRA during training—this trains the adapter on top of a quantized base model, so the adapter learns to compensate for quantization artifacts. The resulting adapter will perform better when served on a quantized base. QLoRA is supported by most fine-tuning libraries, including the Hugging Face peft library and axolotl.

By the end of this step, you should have concrete data comparing quantized and non-quantized serving. You'll know whether quantization is viable for your use case, and you'll have a clear picture of the tradeoffs: how much memory you save, how much latency you gain or lose, and whether quality remains acceptable. This data becomes the foundation for your deployment decision in the next step.