Tuning Large Language Models for Real-World ApplicationsChapter 143

Step 3: Serve Base Model + LoRA with vLLM

Section 3 of 8-~ 5 min read-Synced from Cuantum content

Now that you've verified your model artifacts and installed the necessary dependencies, it's time to actually serve your model. This is where vLLM comes in—it's an inference server specifically optimized for large language models, and it includes native support for LoRA adapters.

The key insight here is that vLLM lets you load the base model once and then dynamically attach different LoRA adapters. This is powerful because it means you can serve multiple task-specific behaviors (different adapters) without having to load multiple copies of the full base model into memory. In a production setting, this can save gigabytes of GPU memory and let you serve more traffic on the same hardware.

vLLM provides an OpenAI-compatible API server, which means you can interact with it using the same request format you'd use with OpenAI's ChatGPT API. This makes integration straightforward—if you've written code that talks to OpenAI's endpoints, you can point it at your vLLM server with minimal changes.

Here's the command to start the server:

python -m vllm.entrypoints.openai.api_server \  --model mistralai/Mistral-7B-Instruct-v0.2 \  --port 8000 \  --enable-lora \  --lora-modules mylora=./outputs/lora_adapter \  --max-model-len 4096

Let's break down what each flag does and why it matters:

  • --model: This specifies the base model checkpoint. You can use a Hugging Face model identifier (like mistralai/Mistral-7B-Instruct-v0.2), and vLLM will download it automatically if it's not already cached locally. Alternatively, you can point to a local directory if you've already downloaded the model. The base model contains all the pre-trained knowledge and parameters—this is the heavy lifting.
  • --port 8000: This sets the port where the server will listen for HTTP requests. You can change this to any available port. The server will expose endpoints at http://localhost:8000 (or whatever port you choose).
  • --enable-lora: This flag tells vLLM to activate LoRA support. Without this, vLLM will only serve the base model and ignore any adapter configurations. This is what unlocks the ability to serve fine-tuned adapters efficiently.
  • --lora-modules mylora=./outputs/lora_adapter: This is where you specify your adapter. The syntax is name=path. You're giving your adapter a name (mylora in this example) and pointing to the directory where the adapter weights are stored. The name is important—you'll reference it in your API requests to tell vLLM which adapter to use for each inference call. If you have multiple adapters, you can load them all at startup by repeating this flag with different names and paths.
  • --max-model-len 4096: This caps the maximum context length (input + output tokens combined) that the model will process. Setting this to a reasonable value helps control memory usage. If you try to process very long contexts, GPU memory consumption can spike and cause out-of-memory errors. By capping it, you trade off flexibility (you can't handle ultra-long documents) for stability and predictable resource usage. Adjust this based on your use case—if you're doing summarization of long documents, you might need 8192 or more, but if you're doing short-form Q&A, 2048 might be enough.

Once you run this command, vLLM will start loading the model into GPU memory. You'll see log output showing progress—loading the base model, loading the adapter, initializing the inference engine. This can take 30 seconds to a few minutes depending on your hardware and the model size. When it's ready, you'll see a message indicating that the server is listening on the specified port.

At that point, your server is live and ready to accept requests. The main endpoint you'll interact with is:

  • http://localhost:8000/v1/chat/completions

This endpoint follows the OpenAI Chat Completions API format, which means you send a JSON payload with a messages array (conversation history) and get back a generated response. The compatibility is deliberate—it makes vLLM a drop-in replacement for OpenAI's API in many cases, which simplifies migration and testing.

One thing to note: vLLM handles batching automatically under the hood. If multiple requests arrive at roughly the same time, vLLM will batch them together to maximize GPU utilization. This is one of the reasons vLLM is so much faster than naive inference loops—it's doing smart scheduling and memory management that would be complex to implement yourself. You don't need to do anything special to enable this; it just works.