Step 3: Load Mistral with QLoRA Configuration

We will use 4-bit quantization for memory efficiency.

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig model_name = "mistralai/Mistral-7B-v0.1" bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4") tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(    model_name,    quantization_config=bnb_config,    device_map="auto") if tokenizer.pad_token is None:    tokenizer.pad_token = tokenizer.eos_token

COPY

Code Breakdown

• What you are doing in this step
• You are loading the base Mistral-7B model in a way that makes fine-tuning feasible on limited VRAM.
• In QLoRA, the base model weights are kept quantized (4-bit) to save memory, while the trainable LoRA adapter weights (added in Step 4) are kept in higher precision.
• BitsAndBytesConfig(...): the QLoRA / 4-bit setup
• load_in_4bit=True
• Loads the model weights in 4-bit format, which drastically reduces VRAM usage compared to FP16/FP32 weights.
• bnb_4bit_quant_type="nf4"
• Uses NormalFloat4 (NF4), a quantization scheme that tends to preserve model quality better than naive 4-bit quantization.
• bnb_4bit_use_double_quant=True
• Enables double quantization, which further compresses some quantization constants to save additional memory.
• bnb_4bit_compute_dtype=torch.float16
• Sets the compute type used during forward passes to FP16.
• This is a common default that balances speed and memory usage.
• Tokenizer loading
• tokenizer = AutoTokenizer.from_pretrained(model_name) loads the tokenizer that matches the base model.
• Using the correct tokenizer is essential because tokenization affects sequence length, truncation behavior, and ultimately training stability.
• Loading the model with quantization
• AutoModelForCausalLM.from_pretrained(..., quantization_config=bnb_config, device_map="auto") loads Mistral with your 4-bit configuration.
• device_map="auto" asks Transformers to place model shards on available devices automatically.
• This is convenient for single-GPU setups, and it can also help when you have multiple GPUs.
• Padding token fix (small but important)
• Some decoder-only models do not define a pad_token by default.
• Setting tokenizer.pad_token = tokenizer.eos_token prevents padding-related issues later (for example during batching, collation, or trainer setup).

At this point, you have an efficient 4-bit base model loaded and ready. In the next step, you will attach LoRA adapters so that only a small number of parameters become trainable.

Why QLoRA here

Mistral-7B normally requires significant VRAM.

With QLoRA, you can fine-tune it on a 24GB GPU — sometimes even less.