Step 7: Compare Results and Make a Deployment Decision

At this point, you've completed the measurement phase of the project. You should have three key artifacts in front of you:

• A running vLLM server configured to serve your base model with a LoRA adapter loaded
• Request logs containing latency measurements, token counts, and GPU memory usage for a representative sample of inference requests
• A quantized test run—either a full vLLM deployment with a quantized checkpoint, or a local comparison using bitsandbytes that gives you a rough sense of memory savings and quality impact

Now comes the interpretation phase. This is where you stop being a data collector and start being a deployment engineer. You're going to look at the numbers you've gathered and make concrete decisions about what configuration to use in production. The goal isn't to find the "perfect" setup—it's to find a setup that meets your requirements while staying within your constraints.

Start by defining your constraints explicitly. What is the maximum acceptable latency for your application? Is this an interactive chatbot where users expect sub-second responses, or a batch processing pipeline where 5-10 seconds per request is fine? What GPU hardware do you have access to? Are you deploying on a single A100 with 40GB of VRAM, or a cluster of smaller GPUs? What's your budget for inference—are you paying per GPU-hour in the cloud, or running on-prem hardware with fixed costs? And finally, what's your quality threshold? Can you tolerate a 2% drop in task accuracy if it cuts your costs in half, or is this a mission-critical application where even tiny quality degradations are unacceptable?

Once you've written down your constraints, compare them against your measurements. Look at your latency distribution: what's the median (p50) latency? What's the 95th percentile (p95)? The p95 is particularly important because it tells you what your worst-case users experience under normal load. If your p95 latency is 3 seconds but your requirement is 1 second, you have a problem. Similarly, look at your GPU memory usage. What's the peak memory consumption? If you're hitting 38GB on a 40GB GPU, you have almost no headroom for traffic spikes or longer input sequences—that's a risk.

Now let's walk through the most common performance issues you'll encounter and how to address them systematically.

If latency is too high

High latency usually comes from one of three sources: insufficient batching, inefficient GPU utilization, or an overly large model for your use case.

The first thing to check is whether vLLM is batching requests effectively. vLLM is designed to handle continuous batching—it groups multiple concurrent requests together and processes them in parallel, which amortizes the cost of model loading and memory transfers. But if your traffic pattern consists of isolated, sequential requests with no concurrency, batching doesn't help. You can simulate concurrent load in your monitoring script by sending multiple requests in parallel using asyncio or threading. If latency drops significantly when you introduce concurrency, that tells you batching is working and you just need more concurrent traffic to see the benefit in production.

If batching isn't the issue, the next lever is reducing the amount of computation per request. The two most impactful parameters here are max_tokens (the maximum number of output tokens per request) and max_model_len (the maximum total sequence length, including input and output). Generating 500 tokens takes roughly five times as long as generating 100 tokens, because each token requires a full forward pass through the model. If your application doesn't need long outputs—maybe you're generating short summaries or single-sentence responses—cap max_tokens aggressively. Similarly, if your users rarely send long prompts, you can reduce max_model_len to free up memory for larger batches, which improves throughput and reduces per-request latency.

If you've tuned batching and output length and latency is still unacceptable, you're likely model-bound. This means the model itself is too large for your latency budget. At this point, you have two options: switch to a smaller model, or use distillation. Switching to a smaller model is straightforward—if you're currently serving Mistral-7B, try Mistral-7B-Instruct or even a 3B variant if one exists in your model family. You'll sacrifice some capability, but you'll gain speed. Distillation is more sophisticated: you train a smaller "student" model to mimic the behavior of your larger "teacher" model. This preserves more of the original quality than simply switching to a smaller pretrained model, but it requires an additional training step. For many applications, distillation is worth the effort if you're deploying at scale.

If GPU memory is too high

Memory issues are easier to diagnose than latency issues because the constraints are hard: if you run out of VRAM, your server crashes. But even if you're not crashing, running close to your memory limit is dangerous—it leaves no room for traffic spikes, larger-than-usual inputs, or KV cache growth when handling long conversations.

The first thing to try is reducing max_model_len. The KV cache—the memory used to store attention keys and values for all previous tokens in a sequence—grows linearly with sequence length. A 7B model with a context window of 4096 tokens uses roughly 2-3GB of KV cache memory per request. If you reduce the context window to 2048 tokens, you cut that memory usage in half. This is a clean win if your application doesn't need long context. For example, if you're building a customer support chatbot that handles short, isolated questions, a 2048-token context is more than sufficient.

The second option is quantization, which we've already covered in detail. If you haven't tested quantization yet, now is the time. A 4-bit quantized model uses roughly one-quarter the memory of a full-precision model, which means you can fit a 7B model on a GPU that would otherwise only support a 1.5B model. This is often the single most impactful optimization for memory-constrained deployments.

If quantization and context length reduction aren't enough, consider switching to a smaller base model. A 3B model uses roughly half the memory of a 7B model, and for many tasks—especially after fine-tuning—the quality gap is smaller than you'd expect. The best way to find out is to fine-tune both models on your task and compare their performance on a held-out test set.

Finally, if you're running into memory limits because of high concurrency—for example, you're trying to serve 50 concurrent requests and running out of memory for KV cache—you can reduce concurrency by tuning vLLM's max_num_seqs parameter. This caps the number of requests processed in parallel. The tradeoff is that additional requests will queue, which increases latency for those requests, but it prevents out-of-memory crashes.

If quality drops too much after quantization

Quality degradation from quantization is usually subtle, but occasionally it's severe enough to be unacceptable. This happens most often with smaller models (quantizing a 3B model hurts more than quantizing a 13B model, because smaller models have less redundancy) or with tasks that require precise numerical reasoning or factual recall.

If you've tested 4-bit quantization and quality is unacceptable, try a different quantization method. AWQ and GPTQ use different algorithms to decide which weights to quantize and how to round them. AWQ tends to preserve quality better for instruction-following tasks, while GPTQ is sometimes better for perplexity-sensitive tasks like language modeling. The only way to know which works better for your specific model and task is to test both.

Another option is to train your LoRA adapter using QLoRA. As mentioned earlier, QLoRA trains the adapter on top of a quantized base model, so the adapter learns to compensate for quantization artifacts. If you originally trained your adapter on a full-precision model and then applied it to a quantized base at serving time, the mismatch can hurt quality. Retraining with QLoRA eliminates this mismatch and often recovers most of the lost quality.

If neither of those approaches works, you might need to accept that quantization isn't viable for your task, and instead pursue distillation. Distillation trains a smaller model to mimic a larger one, preserving quality better than quantization while still reducing memory and latency. It requires more upfront effort—you need to run inference on a large dataset with your teacher model and then train a student model on those outputs—but for high-stakes applications, it's often the best path forward.