Tuning Large Language Models for Real-World ApplicationsChapter 42

2.2 Hugging Face PEFT and TRL libraries in practice

Section 2 of 5-~ 48 min read-Synced from Cuantum content

In the previous section, you learned the theory behind LoRA, QLoRA, Adapters, BitFit, and Prefix Tuning. You explored how each method reduces the number of trainable parameters while preserving the model's ability to adapt to new tasks. You understood the mathematical foundations, architectural choices, and trade-offs between memory efficiency, expressiveness, and deployment flexibility.

Now it's time to move from conceptual understanding to hands-on implementation. Theory provides the map, but implementation is where you learn to navigate the terrain.

In practice, most modern PEFT workflows rely on two essential libraries from Hugging Face:

  • PEFT (Parameter-Efficient Fine-Tuning) – This library provides a unified interface for attaching and managing adapters such as LoRA, QLoRA, Prefix Tuning, and others. It handles the complex details of injecting trainable parameters into frozen base models, managing gradient flows, and saving/loading adapters modularly. PEFT abstracts away the low-level implementation so you can focus on configuration and experimentation.
  • TRL (Transformer Reinforcement Learning) – Originally designed for reinforcement learning from human feedback (RLHF), TRL has evolved into a powerful toolkit for fine-tuning language models. Its SFTTrainer (Supervised Fine-Tuning Trainer) is particularly convenient for PEFT workflows. It handles dataset formatting, tokenization, training loops, and checkpointing with minimal boilerplate, making it easy to integrate with PEFT-enabled models.

Together, these libraries allow you to fine-tune large models efficiently, cleanly, and with minimal boilerplate. You don't need to manually implement LoRA matrix multiplication or write custom training loops. Instead, you configure the adapter, attach it to your model, and let the libraries handle the rest.

This combination of PEFT and TRL has become the de facto standard for parameter-efficient fine-tuning in the open-source ecosystem. It's what practitioners use in production, and it's what you'll learn to use fluently in this section.

In this section, you will:

  • Load a base model with quantization (QLoRA-ready setup) to simulate realistic hardware constraints and learn how to work with models that would otherwise exceed your GPU memory.
  • Attach LoRA adapters using PEFT's configuration system, specifying which layers to target, what rank to use, and how to balance efficiency with expressiveness.
  • Fine-tune using SFTTrainer, leveraging TRL's optimized training loop that handles gradient accumulation, mixed precision, and logging automatically.
  • Save and reload adapters as modular artifacts, learning how to manage multiple task-specific adapters without duplicating the base model.
  • Run inference with PEFT-enabled models, understanding how to generate text with fine-tuned adapters and compare outputs before and after fine-tuning.

By the end of this section, you will have built a complete PEFT pipeline from scratch. You will understand not just the theory, but the practical steps required to deploy parameter-efficient fine-tuning in real projects.

Let's build this step by step.

2.2.1 Setting Up the Environment (Minimal + Reproducible)

Install the core stack:

pip install -U transformers datasets accelerate peft trl bitsandbytes

What you’re installing (in one line each):

  • transformers: models, tokenizers, generation, Trainer plumbing
  • datasets: fast dataset loading + caching
  • accelerate: device placement, mixed precision, multi-GPU ergonomics
  • peft: LoRA, QLoRA, prefix tuning, adapter plumbing
  • trl: SFTTrainer for supervised fine-tuning workflows
  • bitsandbytes: 4-bit / 8-bit quantization for VRAM-bound setups

Configure Accelerate once per machine:

accelerate config
Engineering default: store your final accelerate config file alongside the project, or at least document it in your run notes.

Reproducibility baseline (worth doing up front)

  • Pin key versions (especially transformers, peft, trl, bitsandbytes).
  • Record GPU + CUDA driver versions.
  • Fix random seeds in your training script.
  • Write outputs to a run folder that includes the config you used.

You can skip all of this for a toy demo. You cannot skip it for work you intend to trust.

2.2.2 Loading a Model with QLoRA Configuration (Load Like You’ll Debug It)

The point of this step is not just “make it run.” It is to load the base model in a way that is repeatable, inspectable, and consistent with how you will reload it for inference later.

We’ll load the base model in 4-bit (QLoRA-ready) to simulate real VRAM constraints.

Complete minimal load:

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.float16,    bnb_4bit_use_double_quant=True,    bnb_4bit_quant_type="nf4") tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(    model_name,    quantization_config=bnb_config,    device_map="auto") if tokenizer.pad_token is None:    tokenizer.pad_token = tokenizer.eos_token

Understanding the Quantization Configuration

Let's break down each component of the BitsAndBytesConfig to understand what's happening under the hood:

loadin4bit=True

This parameter tells the library to load the model weights using 4-bit precision instead of the standard 16-bit or 32-bit precision. In practice, this means each weight parameter is represented using only 4 bits of information instead of 16 or 32 bits, resulting in a memory footprint that is roughly 1/4 or 1/8 of the original size. This dramatic reduction in memory usage is what makes it possible to load and fine-tune models that would otherwise be impossible to fit on your GPU.

bnb4bitcompute_dtype=torch.float16

While the weights are stored in 4-bit format, the actual computations during forward and backward passes are performed in 16-bit floating point precision (float16). This is a crucial distinction: quantization reduces storage requirements, but computations still happen at higher precision to maintain numerical stability and training quality. The weights are temporarily dequantized to float16 when needed for computation, then the results are stored back in 4-bit format.

bnb4bitusedoublequant=True

This enables "double quantization," which is a nested quantization technique that further reduces memory usage. In addition to quantizing the model weights themselves, this also quantizes the quantization constants (the scaling factors used in the quantization process). While this sounds recursive, it provides an additional memory reduction of approximately 0.4 bits per parameter on average, which can be significant for very large models.

bnb4bitquant_type="nf4"

The "nf4" quantization type stands for "NormalFloat4," a data type specifically designed for neural network weights. Unlike uniform quantization schemes that divide the value range into equal bins, NF4 uses a non-uniform distribution that is optimized for the typical distribution of neural network weights, which tend to follow a normal (Gaussian) distribution. This specialized quantization scheme preserves model quality better than naive 4-bit quantization would.

Loading the Model and Tokenizer

After configuring quantization, we load both the tokenizer and the model. The device_map="auto" parameter automatically handles device placement, distributing model layers across available GPUs or falling back to CPU when necessary. This is particularly useful when working with models that are large enough to require multiple GPUs or CPU offloading.

The final step checks whether the tokenizer has a padding token defined, and if not, sets it to the end-of-sequence token. This is necessary because many pre-trained models don't define a padding token by default, but padding is essential during batch training to ensure all sequences in a batch have the same length.

Why this matters

  • load_in_4bit=True reduces memory usage drastically—typically by 75% compared to 16-bit precision. A model that would normally require 16GB of VRAM might now fit in just 4GB, making it accessible on much more affordable hardware.
  • nf4 quantization is optimized for training, not just inference. Unlike some quantization schemes that work well for inference but degrade during training, NF4 maintains gradient quality well enough to support effective fine-tuning. This is crucial because you're not just loading the model to generate text—you're going to update it through backpropagation.
  • You can now train models much larger than your GPU would normally allow. With QLoRA, practitioners have successfully fine-tuned 65B parameter models on consumer GPUs with 24GB of VRAM—something that would typically require multiple high-end datacenter GPUs. This democratization of access to large model fine-tuning is one of the most significant practical advances in recent years.

At this point, the base model is frozen and quantized. All the model's original weights are loaded in 4-bit precision and are not trainable. The model can generate text, but it hasn't been adapted to your specific task yet. Memory usage is minimal, and you have plenty of room left for the additional components needed for training: optimizer states, gradients, and activations.

Now we add LoRA.

2.2.3 Attaching LoRA Adapters with PEFT (Target Modules + Rank, Like an Engineer)

Now that the base model is loaded (often quantized), LoRA is where you make the first real engineering choices.

Most “beginner” examples treat r=8 and target_modules=["q_proj","v_proj"] as magic constants.

At intermediate level, you want to understand what you are buying when you change them.

Minimal baseline (works on many decoder-only LLMs)

from peft import LoraConfig, get_peft_model lora_config = LoraConfig(	r=8,	lora_alpha=16,	target_modules=["q_proj", "v_proj"],	lora_dropout=0.1,	bias="none",	task_type="CAUSAL_LM",) model = get_peft_model(model, lora_config)model.print_trainable_parameters()

The two knobs that matter most

1) target_modules: where you allow the model to change

LoRA is only applied to the modules you name. This is not cosmetic. It defines the “control surface” of adaptation.

Common attention targets (decoder-only transformer):

  • q_proj and v_proj are a strong default because they directly change attention behavior.
  • Adding k_proj and o_proj increases capacity, but also increases trainable parameters.

A practical progression that scales cleanly:

  • Step 1 (default): target_modules=["q_proj","v_proj"]
  • Step 2 (more capacity): target_modules=["q_proj","k_proj","v_proj","o_proj"]
  • Step 3 (task needs it): include MLP projections if the architecture exposes them (often names like gate_proj, up_proj, down_proj).
Names are model-specific. Some models do not use qproj/vproj naming. If PEFT errors with “module not found,” inspect module names and update target_modules to match the architecture.

Rule of thumb:

  • If the task is mostly style + formatting + instruction following, attention-only targets are often enough.
  • If the task needs domain reasoning shifts or more “new behavior,” expand targets (attention + some MLP) before jumping to full SFT.

2) r (rank): how much capacity you give the adapters

Rank r controls adapter capacity. Bigger r means:

  • more trainable parameters,
  • more optimizer state,
  • more VRAM and compute,
  • usually easier learning (up to a point).

Practical rank guidance (7B/13B mindset):

  • Start: r=8 (fast, cheap, often surprisingly strong)
  • If underfitting: move to r=16
  • If still underfitting: try r=32 or broaden target_modules (often the better next move)

A useful way to think about it:

  • Increasing rank deepens adaptation in the same places.
  • Increasing target modules broadens adaptation to more places.

Secondary knobs (important, but not the first thing to change)

lora_alpha: strength / scaling

The effective scale is roughly lora_alpha / r.

  • A common stable convention is lora_alpha = 2 * r.
  • If training feels unstable or the adapter effect is too aggressive, reduce lora_alpha before changing everything else.

lora_dropout

  • Defaults like 0.05–0.1 are reasonable.
  • If you have a tiny dataset and see overfitting, increase it slightly.

bias="none"

Leaving biases frozen is usually a sensible default. If you are doing careful ablations, then test lora_only.

A repeatable tuning workflow (recommended)

  1. Fix the dataset + prompt template first.
  2. Start with q_proj/v_proj, r=8.
  3. If quality plateaus, expand targets to q/k/v/o.
  4. If still underfitting, increase rank to 16 or 32.
  5. Only then consider full SFT.

This is the point where PEFT stops being “10 lines of code” and becomes a controllable system.

Understanding the LoRA Configuration Parameters

Let’s examine each parameter in the LoraConfig to understand what it controls and why these choices matter:

r=8 – The Rank of the Low-Rank Decomposition

This is perhaps the most important hyperparameter in LoRA. The rank r determines the dimensionality of the low-rank matrices that will approximate the weight updates. Remember from our mathematical exploration that instead of learning a full weight update matrix ΔW of dimension d×d, we learn two smaller matrices: A (d×r) and B (r×d), where ΔW ≈ BA.

A rank of 8 means we're using very small adapter matrices. For a typical attention layer where d might be 2048, instead of learning 2048×2048 = 4,194,304 parameters, we learn only (2048×8) + (8×2048) = 32,768 parameters—a reduction of over 99%.

The choice of rank involves a trade-off: lower ranks (like 4 or 8) are more memory-efficient but may have limited capacity to learn complex behavioral changes. Higher ranks (like 64 or 128) can capture more nuanced adaptations but require more memory and computation. In practice, ranks between 8 and 16 work well for most fine-tuning tasks.

lora_alpha=16 – The Scaling Factor

This parameter controls how much the LoRA adapter's contribution is scaled before being added to the frozen base weights. The effective learning rate for the adapter is scaled by lora_alpha / r. With lora_alpha=16 and r=8, the scaling factor is 2.

This scaling helps stabilize training and makes the hyperparameter choices more transferable across different model sizes. A common convention is to set lora_alpha to twice the rank, though this can be adjusted based on your specific needs. Higher values give the adapter more influence, while lower values make its contributions more subtle.

targetmodules=["qproj", "v_proj"] – Which Layers to Adapt

This parameter specifies which linear layers in the model should receive LoRA adapters. In transformer architectures, the attention mechanism consists of four linear projections: query (qproj), key (kproj), value (vproj), and output (oproj).

By targeting only q_proj and v_proj, we're adding adapters to the query and value projections while leaving the key and output projections frozen. This is a common choice that balances effectiveness with efficiency. You could target more modules (like ["q_proj", "k_proj", "v_proj", "o_proj"]) for potentially better performance at the cost of more trainable parameters, or fewer modules to maximize efficiency.

Different models may use different naming conventions for these layers, so you may need to inspect your specific model's architecture to identify the correct layer names. The PEFT library provides utilities to help you discover which modules are available in your model.

lora_dropout=0.1 – Regularization Through Dropout

This applies dropout to the LoRA adapter layers during training, randomly setting 10% of the adapter activations to zero. This acts as a regularization technique that helps prevent overfitting, especially important when fine-tuning on smaller datasets.

The dropout is applied only to the adapter matrices, not to the frozen base model weights. A value of 0.1 (10% dropout) is a reasonable default, though you might increase this to 0.2 or 0.3 if you notice overfitting on very small datasets.

bias="none" – Handling Bias Parameters

This parameter determines whether bias terms in the linear layers should be trainable. Setting it to "none" means all biases remain frozen along with the base model weights. Alternative options include "all" (train all biases) or "lora_only" (only train biases in LoRA layers).

In practice, keeping biases frozen (as we do here) is usually sufficient and further reduces the number of trainable parameters. Training biases typically provides only marginal improvements while increasing memory requirements.

tasktype="CAUSALLM" – The Model's Task Type

This tells PEFT what kind of task your model is designed for. "CAUSAL_LM" indicates causal language modeling—the standard autoregressive task where the model predicts the next token given previous tokens. Other options include "SEQ_2_SEQ_LM" for sequence-to-sequence models, "SEQ_CLS" for sequence classification, and others.

This parameter helps PEFT apply the correct internal configurations and optimizations for your specific use case.

Applying the Configuration to Your Model

The line model = get_peft_model(model, lora_config) is where the magic happens. This function takes your frozen, quantized base model and wraps it with the LoRA adapters you've configured. Internally, this:

  • Identifies all the target modules you specified (qproj and vproj in our case)
  • Creates new low-rank adapter matrices A and B for each targeted layer
  • Initializes these matrices (typically with random values for A and zeros for B, ensuring the adapter starts with zero contribution)
  • Marks only these new adapter parameters as trainable while keeping the base model frozen
  • Modifies the forward pass so that when the model processes input, it computes both the frozen base transformation and the adapter transformation, combining them additively

After this call, your model is fundamentally transformed. The base weights remain untouched and frozen, but you now have small, trainable adapter layers inserted into the attention mechanism.

Verifying the Parameter Efficiency

The final line, model.print_trainable_parameters(), is crucial for verification. It outputs a summary showing the total number of parameters in the model and what percentage of them are trainable.

You should see output similar to:

trainable params: 4,194,304 || all params: 1,100,000,000 || trainable%: 0.38%

This demonstrates the dramatic efficiency gain: you're training less than 1% of the model's parameters. In a full fine-tuning scenario, all 1.1 billion parameters would need gradient computation, storage of optimizer states, and weight updates. With LoRA, only the ~4 million adapter parameters require these computational resources.

What Just Happened – A Complete Summary

  • The base weights remain frozen. Every single parameter from the original pre-trained model—all 1.1 billion of them—remains unchanged throughout training. These weights are stored in 4-bit precision and are never updated.
  • Only low-rank matrices inside attention layers are trainable. The small adapter matrices you configured (with rank 8, targeting query and value projections) are the only parameters that will receive gradient updates during training.
  • You are now performing QLoRA fine-tuning. The combination of quantized base weights (Q) and low-rank adaptation (LoRA) is what makes this QLoRA. You get the memory efficiency of quantization plus the parameter efficiency of LoRA.

The printed output showing a very small percentage of trainable parameters (often below 1%) is not a limitation—it's the entire point. This extreme efficiency is what allows you to fine-tune large models on modest hardware while achieving results comparable to full fine-tuning.

That's the power of PEFT. You've maintained the full capacity of your base model while adding a tiny, trainable adaptation layer that will learn task-specific behaviors. The base model's general knowledge remains intact, while your adapters learn the specific patterns and behaviors you want to instill through fine-tuning.

2.2.4 Preparing the Dataset for TRL

TRL’s SFTTrainer is happiest when your dataset has one column that already contains the exact text you want the model to learn.

If you followed Chapter 1, you likely already have a train.jsonl where each row is a complete instruction + response example stored under a text field.

Example row:

{"text":"### Instruction:\nExplain tokenization.\n### Response:\nTokenization splits text into tokens."}

Load it with datasets:

from datasets import load_dataset dataset = load_dataset("json", data_files="data/train.jsonl", split="train")

Sanity-check before training:

print(dataset)print(dataset[0]["text"])
If your dataset already has a text field with the full formatted example, you do not need a custom data collator or extra preprocessing to start.

Understanding the Dataset Format

TRL's SFTTrainer is designed to work with datasets that contain a text field with your training examples. If you completed Chapter 1, you should already have a train.jsonl file where each line is a JSON object containing your instruction-response pairs formatted as complete text sequences.

Here's what a typical entry looks like:

{"text": "### Instruction:\nExplain tokenization.\n### Response:\nTokenization splits text into tokens."}

Each JSON object contains a single "text" field that includes both the instruction prompt and the expected response. The ### Instruction: and ### Response: markers help the model distinguish between the input context and the target output it should learn to generate.

Loading the Dataset with Hugging Face Datasets

To load this JSONL file into a format that TRL can work with, we use the Hugging Face datasets library:

from datasets import load_dataset dataset = load_dataset("json", data_files="data/train.jsonl", split="train")

Let's break down what each parameter does:

  • "json" specifies the file format. The datasets library will parse each line as a separate JSON object.
  • data_files="data/train.jsonl" points to your training data file. Adjust this path to match where your file is located.
  • split="train" tells the library to load this data as a training split. This is important for compatibility with training APIs that expect named splits.

After running this code, dataset will be a Hugging Face Dataset object—an efficient, memory-mapped data structure that can handle datasets much larger than your available RAM.

Why No Additional Formatting Is Needed

One of the conveniences of TRL's SFTTrainer is that if your dataset already contains a field called "text" with your complete training examples (instruction + response), no additional preprocessing or formatting is required. The trainer will automatically:

  • Read each example from the "text" field
  • Tokenize the text using your model's tokenizer
  • Handle batching and padding during training
  • Apply sequence packing if you enable it (which we'll do in the next section)

This is in contrast to some other training frameworks where you might need to manually apply chat templates, separate prompts from completions, or write custom data collators. TRL simplifies this entire process.

Verifying Your Dataset

Before proceeding to training, it's good practice to inspect your loaded dataset to ensure it contains what you expect:

print(dataset)print(dataset[0])

The first line will show you the dataset structure, including the number of examples and available fields. The second line will print the first training example, allowing you to verify that the "text" field contains properly formatted instruction-response pairs.

If you see your expected format with clear instruction and response sections, you're ready to move on to training. If something looks wrong—perhaps the text is truncated, improperly formatted, or missing—now is the time to revisit your data preparation process from Chapter 1.

2.2.5 Training with TRL’s SFTTrainer (Knobs, a Smoke Run, and Evaluation Discipline)

Many fine-tuning projects fail in the same way: the code runs, the loss drops, and you still can’t tell whether you trained something useful.

The fix is boring but effective:

  • Learn the small set of knobs that matter.
  • Run a smoke test.
  • Compare base vs adapter on a fixed prompt suite.

The training knobs that actually move outcomes

Below are the knobs that typically matter most. Try to change one at a time, and write down what you expect to happen before you run.

1) Sequence length (maxseqlength)

This is not just a performance choice. It changes what the model can learn.

  • Too short, and you truncate the exact formatting or reasoning patterns you wanted.
  • Too long, and you pay in memory, instability, and slower iteration.

A practical default is 512–1024 depending on your examples.

2) Effective batch size (micro-batch × gradient accumulation)

With QLoRA, you are usually memory-limited, so you choose a small per_device_train_batch_size and use gradient_accumulation_steps to reach a stable effective batch.

Rule of thumb:

  • Keep the micro-batch as large as your GPU allows.
  • Increase accumulation to reach a stable effective batch before touching other settings.

3) Learning rate (for adapters)

Adapters often tolerate higher learning rates than full SFT, but “higher” is not “infinite.”

  • If the loss spikes, oscillates, or the model collapses into repetitive output, reduce LR.
  • If the model barely changes, LR may be too low or the adapter capacity is too small.

A reasonable starting range for LoRA adapters is often 1e-4 to 3e-4, but treat it as something you validate, not something you assume.

4) Epochs (or max steps)

More epochs is not always better.

  • With small datasets, extra epochs often produce “memorize the template” behavior.
  • Prefer shorter runs and evaluate early.

If your dataset is small, it is often better to train fewer epochs and improve the dataset than to crank epochs up.

5) Packing (packing=True)

Packing increases throughput by concatenating multiple short examples into a single sequence.

  • Good for speed.
  • Risky if your formatting is delicate and you do not want example boundaries to blur.

If you care about strict formatting and clean boundaries, start with packing=False for the first clean baseline, then turn packing on after you confirm outputs remain stable.

A smoke run before the “real” run

Before you commit time and interpret results, do a smoke run that answers one question:

“Does the full pipeline work end-to-end, and do outputs move in the expected direction?”

A good smoke run is intentionally small:

  • 50–200 training examples
  • 20–50 steps (or a fraction of an epoch)
  • Frequent logging
  • No fancy sweeps

Example (small but realistic):

from transformers import TrainingArgumentsfrom trl import SFTTrainer training_args = TrainingArguments(	output_dir="outputs/ch2_peft_lora",	per_device_train_batch_size=1,	gradient_accumulation_steps=8,	learning_rate=2e-4,	logging_steps=5,	save_strategy="no",	max_steps=50,	fp16=True,	report_to="none",) trainer = SFTTrainer(	model=model,	train_dataset=dataset.select(range(200)),	dataset_text_field="text",	tokenizer=tokenizer,	max_seq_length=1024,	packing=False,	args=training_args,) trainer.train()
Pass/fail criteria for the smoke run: - It runs without silent misconfiguration.
  • Trainable parameter count looks correct (adapters only).
  • Loss moves down somewhat.
  • A small prompt suite shows outputs shifting in the expected direction.

Evaluation discipline (simple, but non-negotiable)

You do not need a full benchmark suite to be disciplined. You need consistency.

Step 1: Freeze a tiny prompt suite

Create 10–30 prompts that represent what you actually care about:

  • formatting constraints,
  • refusal boundaries (if relevant),
  • domain terminology,
  • “typical” inputs,
  • a few adversarial or confusing cases.

Step 2: Compare base vs adapter on the same prompts

Run the suite:

  • once on the base model,
  • once with the adapter enabled,
  • using the same generation settings.

Step 3: Track three signals

  • Task success (did the model do the job?)
  • Style/format compliance (did it follow the template?)
  • Regression (did it get worse on anything that mattered?)

If you do nothing else, do this.

A training loop you can trust

Once the smoke run passes, scale up:

  • increase examples,
  • increase steps or epochs modestly
  • turn save_strategy="epoch" back on
  • consider packing=True after you confirm formatting is stable

The goal is not to “train longer.” The goal is to learn faster with controlled changes and repeatable evaluation.

Setting Up Training Arguments

First, we need to configure the training process using Hugging Face's TrainingArguments. This object controls every aspect of how training will proceed:

from transformers import TrainingArgumentsfrom trl import SFTTrainer training_args = TrainingArguments(    output_dir="outputs/ch2_peft_lora",    num_train_epochs=3,    per_device_train_batch_size=2,    gradient_accumulation_steps=8,    learning_rate=2e-4,    logging_steps=25,    save_strategy="epoch",    fp16=True,    report_to="none")

Let's examine each parameter and understand why it matters for PEFT training:

  • output_dir="outputs/ch2_peft_lora" specifies where checkpoints, logs, and the final adapter weights will be saved. This directory will be created if it doesn't exist.
  • num_train_epochs=3 means the model will see the entire training dataset three times. For PEFT, you often need fewer epochs than full fine-tuning because you're updating fewer parameters, which can help prevent overfitting.
  • per_device_train_batch_size=2 sets how many examples are processed simultaneously on each GPU. With QLoRA, you can often use smaller batch sizes due to memory constraints, but this is still effective when combined with gradient accumulation.
  • gradient_accumulation_steps=8 is crucial for memory efficiency. Instead of updating weights after every 2 examples, gradients are accumulated over 8 steps (2 × 8 = 16 effective batch size) before performing an update. This simulates training with a larger batch size without the memory requirements.
  • learning_rate=2e-4 (0.0002) is typically higher than what you'd use for full fine-tuning. Because LoRA adapters start from a zero-initialized state, they can tolerate and often benefit from higher learning rates to learn task-specific patterns quickly.
  • logging_steps=25 determines how frequently training metrics are logged. Every 25 steps, you'll see loss values and other statistics.
  • save_strategy="epoch" tells the trainer to save a checkpoint after each complete pass through the dataset. This gives you three checkpoints (one per epoch) that you can compare later.
  • fp16=True enables mixed-precision training using 16-bit floating point numbers where possible. This reduces memory usage and speeds up training, especially on modern GPUs with tensor cores.
  • report_to="none" disables automatic logging to external services like Weights & Biases or TensorBoard. Set this to "tensorboard" or "wandb" if you want to track experiments.

Initializing the SFTTrainer

With our training arguments configured, we can now create the trainer object that will handle the actual training process:

trainer = SFTTrainer(    model=model,    train_dataset=dataset,    dataset_text_field="text",    tokenizer=tokenizer,    max_seq_length=1024,    packing=True,    args=training_args)

The SFTTrainer is specifically designed for supervised fine-tuning and includes several optimizations that make PEFT training more efficient:

  • model=model is your PEFT-wrapped, quantized model with LoRA adapters attached. The trainer automatically detects that this is a PEFT model and will only compute gradients for the adapter parameters.
  • train_dataset=dataset is the Hugging Face dataset we loaded in the previous section.
  • dataset_text_field="text" tells the trainer which field in your dataset contains the training examples. This should match the field name you used when preparing your data.
  • tokenizer=tokenizer provides the tokenizer for converting text into token IDs. The trainer will use this automatically during the training loop.
  • max_seq_length=1024 sets the maximum length for training sequences. Examples longer than this will be truncated, and shorter ones will be padded. This value should match your model's context window and your task requirements.
  • packing=True is an important optimization that concatenates multiple short examples into single sequences up to max_seq_length. This dramatically improves training efficiency by reducing padding waste, especially important when your dataset contains examples of varying lengths.
  • args=training_args passes in all the training configuration we defined earlier.

Starting the Training Process

Finally, we initiate training with a single command:

trainer.train()

This line triggers the entire training loop. The trainer will:

  • Load batches of examples from your dataset
  • Tokenize the text and create input tensors
  • Run forward passes through the model (base weights + LoRA adapters)
  • Calculate loss by comparing predictions to target tokens
  • Compute gradients, but only for the LoRA adapter parameters
  • Accumulate gradients over the specified number of steps
  • Update adapter weights using the optimizer
  • Log metrics and save checkpoints according to your configuration

During training, you'll see output showing the progress, loss values, and training speed. The loss should generally decrease over time, indicating that your adapters are learning the patterns in your dataset.

Important Observation: The Simplicity Is Deceptive

Here's something crucial to understand: if you completed Chapter 1, this training code looks almost identical to what you used for full supervised fine-tuning. The structure is the same. The API calls are the same. The workflow is familiar.

But the underlying computation is fundamentally different:

  • In full fine-tuning: Every single parameter in the model receives gradient updates. All 1+ billion weights are modified during training. Optimizer states (momentum, variance) must be stored for every parameter. Memory requirements scale with the model size.
  • In PEFT with LoRA: Only the tiny adapter matrices receive gradient updates. The base model's 1+ billion parameters remain completely frozen—they're never modified, and no optimizer states are stored for them. Gradients are only computed for the ~4 million adapter parameters. Memory requirements are dramatically reduced.

This is the power of PEFT: you keep the familiar training workflow that you already understand, but you gain enormous efficiency improvements under the hood. You don't need to learn a completely new training paradigm or restructure your code. You simply configure PEFT adapters, and the rest of the training process remains intuitive and accessible.

The efficiency gains are not just incremental—they're transformative. You can now fine-tune models that would otherwise be impossible to train on your hardware. You can iterate faster because training completes more quickly. You can experiment with more hyperparameters because each training run consumes fewer resources.

And perhaps most importantly: the quality of the fine-tuned model is typically comparable to full fine-tuning for most tasks. You're not sacrificing capability for efficiency—you're achieving both.

2.2.6 Saving and Reloading LoRA Adapters (Artifacts + Reproducibility)

PEFT’s killer feature is that your trained artifact is small and modular.

But “small” is not the same as “reproducible.” Treat adapters like a real deliverable: save the weights, save the config, and save enough metadata to reload the run later without guesswork.

Saving Your Trained Adapters

After training completes, you can save your LoRA adapters with just two simple commands:

trainer.model.save_pretrained("outputs/ch2_peft_lora/final")tokenizer.save_pretrained("outputs/ch2_peft_lora/final")

The first line saves only the adapter weights. The base model is not duplicated.

The second line saves the tokenizer config. It often won’t change, but saving it prevents “works on my machine” reload failures later.

What to store with every adapter (practical): - base model identifier (exact repo + revision/commit if possible)
  • quantization config (4-bit / nf4 / compute dtype)
  • LoRA config (r, alpha, target_modules, dropout)
  • training args (LR, seq len, batch × accumulation, max steps)
  • data version (hash of train.jsonl or dataset commit)
  • a small prompt suite used for base vs adapter comparison

Understanding What Gets Saved

When you call save_pretrained() on a PEFT model, the library intelligently detects that only adapter weights need to be saved. The saved directory will contain:

  • An adapter_config.json file specifying the LoRA configuration (rank, alpha, target modules, etc.)
  • An adapter_model.bin or adapter_model.safetensors file containing the actual trained adapter weights
  • Tokenizer files if you saved the tokenizer to the same directory

This is dramatically different from full fine-tuning, where you would save the complete model state including all billions of parameters, optimizer states, and training metadata.

Reloading Your Adapters

Later, when you want to use your fine-tuned model, you reload it in two stages. First, you load the base model exactly as you did before training:

from peft import PeftModel base_model = AutoModelForCausalLM.from_pretrained(    model_name,    quantization_config=bnb_config,    device_map="auto")

This loads the original, unmodified base model from Hugging Face. If you're using quantization (as in QLoRA), you apply the same quantization configuration here. The base model loads in its original state, completely unaware of any fine-tuning.

Second, you attach your trained adapters to this base model:

model = PeftModel.from_pretrained(    base_model,    "outputs/ch2_peft_lora/final")

The PeftModel.from_pretrained() method takes your base model and loads the adapter weights from the specified directory. It reads the adapter_config.json to understand the LoRA architecture, then loads the trained weights from adapter_model.bin. Finally, it injects these adapters into the appropriate layers of the base model, recreating the exact same model architecture you had after training.

What You Have After Reloading

After these two steps, your model variable contains:

  • The frozen base model weights (loaded from Hugging Face or your local cache)
  • Your trained LoRA adapter matrices attached to the target layers
  • The same computational behavior you had at the end of training

When you run inference, the model will automatically route computations through both the base weights and the adapters, producing outputs that reflect your fine-tuning.

The Power of Modularity

This two-stage loading process unlocks powerful capabilities that aren't possible with full fine-tuning:

  • Maintain multiple adapters for different tasks: You can train separate adapters for customer support, technical documentation, creative writing, or any other task. Each adapter remains a small, independent file. When you need a specific behavior, you simply load that adapter onto the base model.
  • Switch behaviors without duplicating large models: Instead of storing five complete 10GB models for five different tasks (50GB total), you store one 10GB base model and five 50MB adapters (10.25GB total). When you want to switch tasks, you don't need to load an entirely new model—you just swap out the adapter, which takes seconds rather than minutes.
  • Share and version control efficiently: Because adapters are small, you can easily share them with colleagues, upload them to model hubs, or track them in version control systems like Git. This makes experimentation and collaboration much more practical.
  • Serve multiple models simultaneously: In production environments, you can load the base model once into memory, then serve requests for different tasks by dynamically attaching the appropriate adapter. This dramatically reduces memory requirements compared to loading separate full models for each task.

This modularity represents a fundamental shift in how we think about model customization. Instead of creating monolithic, task-specific models, you create a library of lightweight, composable adapters that can be mixed, matched, and deployed as needed.

2.2.7 Inference with a PEFT Model

Once you've trained your PEFT model and saved the adapters, you're ready to use it for inference—generating responses to new prompts. The inference process with a PEFT model is essentially identical to inference with any other language model, but it's worth understanding what's happening under the hood and how to structure your code for optimal results.

Creating a Generation Function

We'll define a helper function that handles the complete inference pipeline:

def generate_response(prompt):    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)     with torch.no_grad():        output = model.generate(            **inputs,            max_new_tokens=120,            temperature=0.7,            top_p=0.9        )     return tokenizer.decode(output[0], skip_special_tokens=True) prompt = "### Instruction:\nExplain gradient accumulation simply.\n### Response:\n" print(generate_response(prompt))

Let's break down each component of this function to understand exactly what's happening:

Tokenization and Device Placement

The first line inside the function converts your text prompt into a format the model can process:

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

The tokenizer converts your human-readable text into token IDs—the numeric representations the model actually operates on. The return_tensors="pt" parameter tells the tokenizer to return PyTorch tensors rather than Python lists. The .to(model.device) ensures the input tensors are placed on the same device (CPU or GPU) as your model, which is essential for the computation to work correctly.

Disabling Gradient Computation

The with torch.no_grad(): context manager is crucial for efficient inference:

with torch.no_grad():    output = model.generate(...)

During training, PyTorch tracks all operations to compute gradients for backpropagation. But during inference, you're only doing forward passes—you never need to compute gradients. By wrapping your generation call in torch.no_grad(), you tell PyTorch to skip gradient tracking entirely, which significantly reduces memory usage and speeds up computation.

Generation Parameters

The model.generate() method is where the actual text generation happens. Let's examine each parameter:

  • **inputs unpacks your tokenized prompt, providing the starting point for generation.
  • max_new_tokens=120 limits the response length to 120 new tokens beyond the input prompt. This prevents runaway generation and controls response verbosity. You can adjust this based on your needs—shorter for concise answers, longer for detailed explanations.
  • temperature=0.7 controls randomness in the generation process. Lower values (closer to 0) make the model more deterministic, always choosing the most likely next token. Higher values (approaching 2.0) increase randomness and creativity. A temperature of 0.7 provides a good balance—responses are generally coherent but not completely predictable.
  • top_p=0.9 implements nucleus sampling, which considers only the most probable tokens whose cumulative probability reaches 0.9. This prevents the model from occasionally choosing very unlikely tokens while still maintaining diversity in generation.

Decoding the Output

After generation completes, you have token IDs that need to be converted back to readable text:

return tokenizer.decode(output[0], skip_special_tokens=True)

The tokenizer.decode() method converts token IDs back to a text string. The output[0] selects the first (and only) generated sequence from the batch. The skip_special_tokens=True parameter removes special tokens like padding tokens, beginning-of-sequence tokens, or end-of-sequence tokens from the final output, giving you clean, human-readable text.

Structuring Your Prompt

Notice the format of the example prompt:

prompt = "### Instruction:\nExplain gradient accumulation simply.\n### Response:\n"

This follows the instruction-response format that many fine-tuned models expect. The structure helps the model understand what type of task you're asking it to perform. If you trained your model on a dataset with a specific prompt template (like Alpaca format, ShareGPT format, or a custom format), you should structure your inference prompts to match that same template. Consistency between training and inference formats is crucial for optimal performance.

What Happens During Generation

When you call this function, here's the computational flow:

  • Your prompt is tokenized into a sequence of token IDs.
  • These tokens are fed into the model as the initial context.
  • The model (base weights + LoRA adapters) processes this context and predicts the next token.
  • The predicted token is appended to the sequence.
  • This extended sequence becomes the new context for predicting the next token.
  • This process repeats iteratively until either max_new_tokens is reached or the model generates a stop token.
  • The complete sequence of generated tokens is decoded back into text and returned.

Throughout this entire process, the model is using both the frozen base model parameters and your trained LoRA adapter weights. The adapters modify the model's behavior in subtle but important ways, steering the generation toward the patterns learned from your fine-tuning dataset.

Observing Your Fine-Tuning Results

At this point, your fine-tuned behavior should clearly reflect your dataset. The responses generated by your model should demonstrate the specific patterns, style, tone, or knowledge you emphasized during training. If you fine-tuned on technical documentation, the model should provide more structured, precise explanations. If you trained on conversational data, responses should be more natural and dialogue-like. If your dataset emphasized conciseness, the model should generate shorter, more direct answers.

Compare the outputs from your fine-tuned model with outputs from the base model using the same prompts. The differences reveal what your adapters have learned. If you're not seeing the expected behavior changes, this might indicate issues with your training data quality, insufficient training duration, or hyperparameter choices that need adjustment.

Iterating and Experimenting

This generation function provides a foundation for experimentation. You can adjust the generation parameters to explore different behaviors:

  • Lower the temperature to 0.3 for more deterministic, focused responses.
  • Raise it to 1.0 or higher for more creative, varied outputs.
  • Adjust max_new_tokens based on whether you need brief answers or detailed explanations.
  • Experiment with other generation parameters like top_k, repetition_penalty, or do_sample to fine-tune the generation behavior.

The beauty of PEFT is that you can quickly load different adapters and compare their outputs on the same prompts, helping you understand how different training approaches affect model behavior without the overhead of managing multiple full-sized models.

2.2.8 Comparing PEFT vs Full Fine-Tuning in Practice (7B/13B Reality Check)

If you are working with 7B or 13B models, the constraint is rarely “can I load the model?” The constraint is whether you can afford the training footprint and the operational overhead once you iterate and deploy.

Below is a practical comparison across three axes: training memory, artifacts, and deployment architecture.

1) Training memory: why full SFT becomes expensive at 7B/13B

Full SFT makes every parameter trainable. That has a compounding effect on memory:

  • Weights (stored in FP16/BF16)
  • Gradients (same order of magnitude as the weights)
  • Optimizer state (often 2× the weights for Adam-like optimizers)
  • Activations (depends heavily on sequence length, batch size, and whether you checkpoint)

At 7B/13B, that stack quickly pushes you into tens of GB of VRAM for training, and the exact number can swing widely with context length and batch size. (This is why “it fits in memory” and “it trains” are two different statements.)

PEFT changes the footprint because only the adapters are trainable. QLoRA goes further by keeping the frozen base weights in 4-bit while training the adapter parameters in higher precision. In many common setups, this is what turns “needs multiple high-VRAM GPUs” into “viable on one GPU,” especially once you add gradient checkpointing and tune sequence length.

2) Artifacts: what you store and what you can iterate on

With full SFT, every variant you produce is a full checkpoint. That means multi‑GB artifacts and slower iteration cycles (save, upload, download, rollback).

With PEFT, most runs produce small adapter artifacts (often tens of MB, depending on rank and target modules). This changes day-to-day workflow:

  • You can keep many variants without exploding storage.
  • You can version adapters more realistically.
  • You can A/B test and roll back quickly.

3) Deployment architecture: one model per behavior vs one base model + adapters

Full SFT tends to push you toward one full model per behavior. At 7B/13B, that becomes heavy fast:

  • More storage for each variant.
  • More VRAM pressure if you want multiple behaviors “hot” at the same time.

PEFT supports a different pattern:

  • Load the base model once.
  • Swap adapters when you need a different behavior.

That architecture is the main reason PEFT shows up so often in production systems that serve multiple tasks, products, or customers.

Practical conclusion

For 7B/13B models, it is usually rational to start with PEFT, measure quality, and only pay the full-SFT cost when you can justify the delta.

2.2.9 When to Choose PEFT vs Full SFT (Decision Rules)

Use PEFT when:

  • You are VRAM-bound or want fast iteration on 7B/13B models.
  • You expect multiple task variants (or customer-specific behaviors).
  • You want small artifacts that are easy to version, ship, and roll back.

Use full SFT when:

  • You can demonstrate you need a large behavioral shift that adapter capacity is not capturing.
  • You are adapting to a highly specialized domain and PEFT variants plateau after reasonable tuning.
  • You can afford the compute and accept slower iteration for a higher ceiling.
Default workflow (practical): start with PEFT → tune data + prompts → raise adapter capacity (rank/targets) if needed → move to full SFT only after you can measure a meaningful quality gap.

2.2.10 Key Takeaway

PEFT does not reduce your control over model behavior. Instead, it fundamentally changes the control surface—the interface through which you shape the model's outputs.

Think of it this way:

  • Full SFT rewrites a model. When you perform full supervised fine-tuning, you're modifying the actual weights throughout the entire network. You're literally changing what the model "knows" at a foundational level. This is powerful, but it's also irreversible without keeping backup checkpoints, and it affects every capability the model has—not just the behavior you're trying to improve.
  • PEFT steers a frozen model through a small set of trainable parameters. With parameter-efficient approaches like LoRA, the base model remains completely unchanged. Your adapters act as lightweight "steering wheels" that redirect the model's existing capabilities toward your desired behavior. The base knowledge stays intact; you're just influencing how it gets expressed.

This distinction has practical implications for how you should think about and work with adapters:

If you treat adapters as modular, testable artifacts—meaning you version them properly, test them against a stable suite of evaluation prompts, and ensure your training runs are reproducible—then PEFT naturally becomes the default path for most intermediate-to-production workflows, especially at the 7B/13B scale where resource constraints matter.

The modularity means you can:

  • Develop multiple specialized behaviors in parallel
  • A/B test different adapter configurations quickly
  • Roll back to previous versions without losing the base model
  • Compose or swap adapters based on context or user needs

In other words, PEFT gives you more operational flexibility even though you're training fewer parameters. The constraint becomes a feature: by forcing you to work within a smaller parameter budget, PEFT encourages cleaner data, better prompts, and more intentional design decisions—all of which tend to produce more maintainable systems in the long run.