Step 5: Add a Lightweight Monitoring Logger

Section 5 of 8-~ 10 min read-Synced from Cuantum content

At this point, you have a working deployment—your server is running, you can send requests, and you're getting responses back. But you're flying blind. You don't have visibility into how the system is actually performing under load, how much GPU memory each request consumes, or whether latency is creeping up over time. In production, this is unacceptable. You need telemetry.

This step introduces a lightweight monitoring logger that captures the metrics that matter most for LLM inference: request latency, response length (as a proxy for token count), and GPU memory usage. These three metrics give you a comprehensive picture of your deployment's health. Latency tells you whether your service is meeting user expectations. Response length helps you understand token throughput and detect anomalies (like unexpectedly long outputs that might indicate a problem with your prompt or model behavior). GPU memory usage is critical for capacity planning—if you're running close to the limit, you'll hit out-of-memory errors as traffic increases.

The script we're about to build is deliberately simple. It's not a replacement for production-grade monitoring systems like Prometheus, Grafana, or Datadog. Instead, it's a starting point—a way to collect structured logs that you can analyze immediately, without requiring infrastructure setup. Once you've validated your deployment and you're ready to scale, you can migrate these metrics into a real monitoring stack. But for prototyping and initial validation, a JSONL log file is fast, portable, and easy to work with.

What we'll measure and log:

Request latency: End-to-end time from when the request is sent to when the response is received. This is the metric your users experience directly.
Response length: Character count of the response, which serves as a rough proxy for token count. While not exact, it's correlated enough to be useful for spotting outliers.
GPU memory usage: Memory consumption before and after each request. This helps you understand memory overhead and detect leaks or inefficiencies.

Create a file called monitor_requests.py with the following code:

import timeimport jsonimport requestsimport psutil try:    import pynvml    pynvml.nvmlInit()    GPU_AVAILABLE = Trueexcept:    GPU_AVAILABLE = False URL = "http://localhost:8000/v1/chat/completions" def gpu_mem_used_mb():    if not GPU_AVAILABLE:        return None    handle = pynvml.nvmlDeviceGetHandleByIndex(0)    mem = pynvml.nvmlDeviceGetMemoryInfo(handle)    return mem.used / (1024**2) def call(prompt, model_name, lora_name=None):    payload = {        "model": model_name,        "messages": [            {"role": "system", "content": "You are a helpful assistant."},            {"role": "user", "content": prompt}        ],        "temperature": 0.2,        "max_tokens": 250    }    if lora_name:        payload["lora"] = lora_name     t0 = time.time()    r = requests.post(URL, json=payload, timeout=60)    t1 = time.time()    r.raise_for_status()     data = r.json()    text = data["choices"][0]["message"]["content"]    return text, (t1 - t0), data def main():    prompts = [        "Summarize DPO in 2 sentences.",        "Write a polite customer support reply: 'My package arrived damaged.'",        "Explain quantization vs distillation with one example each."    ]     logs = []     for p in prompts:        mem_before = gpu_mem_used_mb()        text, latency, raw = call(p, "mistralai/Mistral-7B-Instruct-v0.2", lora_name="mylora")        mem_after = gpu_mem_used_mb()         logs.append({            "prompt": p,            "latency_sec": latency,            "response_chars": len(text),            "gpu_mem_used_mb_before": mem_before,            "gpu_mem_used_mb_after": mem_after,            "timestamp": time.time()        })         print("\nPrompt:", p)        print("Latency:", latency)        print("Chars:", len(text))     with open("outputs/request_logs.jsonl", "a", encoding="utf-8") as f:        for item in logs:            f.write(json.dumps(item) + "\n")     print("\nSaved outputs/request_logs.jsonl") if __name__ == "__main__":    main()

Let's break down what this monitoring script does and why each piece matters:

GPU memory tracking with pynvml: The pynvml library is the Python binding for NVIDIA's management library. It gives you low-level access to GPU metrics without needing to parse command-line output from nvidia-smi. The gpu_mem_used_mb() function queries the GPU for its current memory usage and converts it to megabytes for readability. The try-except block handles environments where GPUs aren't available or the library isn't installed—in those cases, GPU metrics are simply logged as None. This makes the script portable across different development and testing environments.

The call function wraps the HTTP request logic we used earlier, but now it returns not just the response text and latency, but also the full raw response data. This gives you flexibility to extract additional fields later (like token counts, if your vLLM version exposes them in the response). The temperature is set lower here (0.2) compared to the earlier example—this makes outputs more deterministic, which is important when you're benchmarking. You want consistent behavior across runs so you can attribute changes to your configuration, not to sampling randomness.

The test prompts are deliberately varied. They represent different types of tasks: factual summarization, conversational writing, and comparative explanation. This diversity is intentional—different prompt types can have different latency characteristics depending on how the model processes them. By testing a range of tasks, you get a more realistic picture of expected performance. In a real deployment, you'd replace these with prompts drawn from your actual use case—support tickets, customer queries, content generation tasks, whatever reflects your production traffic.

Memory measurement before and after: For each prompt, we capture GPU memory usage immediately before making the request and immediately after receiving the response. The difference tells you how much memory is consumed per request. If you see the "after" value consistently higher than "before," that's a sign of memory accumulation—possibly a leak, or possibly vLLM caching something. If memory stays roughly constant, that's a good sign. If it grows without bound, you have a problem that needs investigation.

Structured logging to JSONL: The script appends each request's metrics to a JSONL (JSON Lines) file. Each line is a complete JSON object representing one request. This format is easy to process with command-line tools like jq, easy to import into pandas for analysis, and easy to stream into monitoring systems. You can run this script multiple times, and new logs will be appended without overwriting previous runs. This is critical for tracking performance over time—you can compare metrics before and after configuration changes, or track how performance degrades as you increase traffic.

Console output for immediate feedback: While the script logs everything to a file, it also prints key metrics to the console. This gives you immediate feedback when running the script interactively. You can spot obvious problems (like a request that takes 30 seconds instead of 3) without having to open the log file.

Run the script to start collecting metrics:

python monitor_requests.py

You'll see output similar to this:

Prompt: Summarize DPO in 2 sentences.Latency: 1.892Chars: 184 Prompt: Write a polite customer support reply: 'My package arrived damaged.'Latency: 2.104Chars: 217 Prompt: Explain quantization vs distillation with one example each.Latency: 2.456Chars: 298 Saved outputs/request_logs.jsonl

The latency values you see will vary based on your hardware, but you should observe some patterns. Longer responses generally take longer to generate (because the model is producing more tokens). The first request in a session might be slower than subsequent ones due to warm-up overhead—vLLM loads the model into memory, initializes its scheduling engine, and sets up GPU kernels. After that, latency should stabilize.

Now open outputs/request_logs.jsonl. Each line is a structured record that looks like this:

{"prompt": "Summarize DPO in 2 sentences.", "latency_sec": 1.892, "response_chars": 184, "gpu_mem_used_mb_before": 14230.5, "gpu_mem_used_mb_after": 14231.2, "timestamp": 1709578320.45}

This is your telemetry baseline. Every optimization you make—quantization, batching, reducing max tokens—should be measured against this data. If you switch to a quantized model and latency drops from 2 seconds to 1.5 seconds, that's a measurable win. If GPU memory usage drops from 14GB to 9GB, that's concrete evidence that quantization is working. Without this data, you're optimizing blind.

A few things to watch for when analyzing your logs:

Latency distribution: Look at the range of latencies across different prompts. If one prompt consistently takes 3x longer than others, investigate why. It might be hitting edge cases in the model, or it might be triggering inefficient token generation patterns.
Memory stability: Compare gpu_mem_used_mb_before and gpu_mem_used_mb_after across multiple runs. If memory keeps climbing, you have a problem. If it stays constant, your deployment is stable.
Correlation between response length and latency: Longer responses should take longer, but the relationship should be roughly linear. If short responses sometimes take as long as long ones, that suggests variability in server load or batching behavior.

This simple logger gives you the foundation for data-driven decision making. As you move through the remaining steps—testing quantization, tuning parameters, preparing for production—you'll keep running this script and analyzing the logs. It becomes your reality check, the source of truth that tells you whether your changes are actually improvements or just noise.