Step 9: Reload and Evaluate

Section 9 of 13-~ 2 min read-Synced from Cuantum content

Reload base + adapter:

from peft import PeftModel base_model = AutoModelForCausalLM.from_pretrained(    model_name,    quantization_config=bnb_config,    device_map="auto") model = PeftModel.from_pretrained(    base_model,    "outputs/ch2_domain_mistral/final")

Code Breakdown

You reload the base model (in the same 4-bit configuration as training).
Then PeftModel.from_pretrained(...) attaches your saved LoRA adapter on top.
This is the practical deployment pattern: one base model, many small adapters.

Test domain-specific prompt:

def generate(prompt):    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)    with torch.no_grad():        output = model.generate(            **inputs,            max_new_tokens=150,            temperature=0.7        )    return tokenizer.decode(output[0], skip_special_tokens=True) prompt = """### Instruction:Write a polite response to a refund request.### Input:Customer says: 'My product arrived damaged.'### Response:""" print(generate(prompt))

Code Breakdown

return_tensors="pt" converts the prompt into PyTorch tensors.
.to(model.device) moves inputs to the same device as the model.
torch.no_grad() disables gradient tracking (faster, less memory).
model.generate(...) produces a completion.
max_new_tokens=150 caps the response length.
temperature=0.7 adds some randomness so outputs are not overly deterministic.
skip_special_tokens=True removes special tokens from the decoded text.

Compare:

Base model output
Fine-tuned model output

You should notice:

More consistent tone
More domain-aligned language
Less generic responses