Step 4: Send Requests with a Simple Client

Section 4 of 8-~ 5 min read-Synced from Cuantum content

Now that your server is running, you need a way to send requests to it and verify that everything works. This step walks you through building a minimal client script that sends a prompt, measures latency, and prints the response.

This might seem trivial, but it's actually a critical piece of infrastructure. Once you have a working client, you can use it as the foundation for more sophisticated testing—load testing, regression testing, A/B comparisons between different models or adapters. It's also a debugging tool: if something goes wrong, you can isolate whether the problem is in the server, the model, or the client logic.

Create a file called client_test.py with the following code:

import timeimport requests URL = "http://localhost:8000/v1/chat/completions" def call_model(user_text, model_name="mistralai/Mistral-7B-Instruct-v0.2", lora="mylora"):    payload = {        "model": model_name,        "messages": [            {"role": "system", "content": "You are a helpful assistant."},            {"role": "user", "content": user_text}        ],        "temperature": 0.7,        "max_tokens": 200    }     # vLLM uses the adapter name via "lora" in some deployments.    # If your setup differs, check vLLM docs for the exact field.    payload["lora"] = lora     t0 = time.time()    r = requests.post(URL, json=payload, timeout=60)    t1 = time.time()     r.raise_for_status()    data = r.json()    text = data["choices"][0]["message"]["content"]     return text, (t1 - t0) if __name__ == "__main__":    prompt = "Explain LoRA in simple terms and give one example use case."    output, latency = call_model(prompt)    print("Latency:", latency)    print("Response:\n", output)

Let's walk through what this code does:

The call_model function is the core of the client. It constructs a request payload in the format that vLLM (and OpenAI) expects. The messages array defines the conversation context. In this case, we're providing a system message (which sets the assistant's behavior) and a user message (the actual prompt). You can extend this to include multi-turn conversations by adding more messages.

The temperature parameter controls randomness in the output. A value of 0.7 is a reasonable default—low enough to be coherent, high enough to avoid repetitive outputs. You can adjust this based on your use case. For factual Q&A, you might want 0.2 or lower. For creative writing, you might go higher.

The max_tokens parameter caps the length of the response. This is important for two reasons: it prevents runaway generation (where the model keeps generating until it hits the model's max length), and it helps you control costs and latency. Longer outputs take more time and use more GPU cycles. Start with a conservative limit and increase it only if you need longer responses.

The payload["lora"] = lora line tells vLLM which adapter to use for this request. This is how you select among multiple adapters if you've loaded more than one. If you omit this field, vLLM will use the base model without any adapter. The exact field name (lora) may vary depending on your vLLM version—check the documentation if you encounter errors. Some versions use adapter or a different key.

Timing the request is done by capturing the current time before and after the HTTP call. This gives you end-to-end latency from the client's perspective, which includes network round-trip time, server processing time, and any queuing if the server is handling multiple requests. This is the latency your users will experience, so it's the metric that matters most for production readiness.

r.raise_for_status() is a safety check—it raises an exception if the server returned an error code (like 500 or 400). This helps you catch configuration problems early. If you see an error here, check the vLLM server logs for details.

Finally, we extract the generated text from the response JSON and return it along with the latency. The response structure follows OpenAI's format: data["choices"][0]["message"]["content"] contains the assistant's reply.

Run the script:

python client_test.py

If everything is configured correctly, you should see output like:

Latency: 2.347Response: LoRA (Low-Rank Adaptation) is a technique for fine-tuning large language models...

The exact latency will depend on your hardware, the model size, and whether this is the first request (which may include warm-up overhead). On a modern GPU like an A100 or 4090, you should see latencies in the range of 1–5 seconds for a 200-token response from a 7B model. If you're seeing much higher latencies, that's a signal to investigate—check GPU utilization, memory usage, and whether the model is actually running on GPU (not CPU).

This baseline measurement is your reference point. Every optimization you make—quantization, batching, reducing max tokens—should be evaluated against this number. If an optimization doesn't improve latency or reduce cost, it's not worth the complexity.