Step 2: Install vLLM and Dependencies

Section 2 of 8-~ 3 min read-Synced from Cuantum content

Now that you know which model artifacts you're working with, you need to set up the runtime environment. This step installs vLLM (the inference server), quantization libraries (if you want to test memory-reduced models), and monitoring tools (so you can measure what's actually happening under the hood).

Start by installing vLLM:

pip install -U vllm

The -U flag upgrades to the latest version, which is important because vLLM is under active development and performance improvements land frequently. If you run into issues, check the vLLM GitHub releases page—sometimes newer versions introduce breaking changes or require updated CUDA drivers.

If you plan to experiment with quantization using bitsandbytes (a common library for 4-bit and 8-bit quantization), install it now:

pip install -U bitsandbytes

Quantization reduces the precision of your model's weights, which lowers memory usage and can speed up inference. The trade-off is that you may lose some quality. Later in this project, you'll measure that trade-off empirically, so you can make an informed decision rather than guessing.

Finally, install a few lightweight monitoring helpers:

pip install -U psutil pynvml requests

Here's what these do:

psutil: Lets you query CPU and memory usage
pynvml: Provides access to NVIDIA GPU metrics (memory usage, temperature, etc.)
requests: A simple HTTP client for sending requests to your vLLM server

You might wonder why we need monitoring tools at this stage. The answer is simple: you can't optimize what you don't measure. Without logging latency, token usage, and GPU memory, you're flying blind. These libraries let you instrument your requests and understand how your system behaves under different configurations—quantized vs. non-quantized, short prompts vs. long prompts, single requests vs. batched requests.

Once these packages are installed, you're ready to start the vLLM server and begin serving requests.