Tuning Large Language Models for Real-World ApplicationsChapter 55

Step 5: Load Dataset

Section 5 of 13-~ 1 min read-Synced from Cuantum content
from datasets import load_dataset dataset = load_dataset(    "json",    data_files="data/domain_train.jsonl",    split="train")

Code Breakdown

  • You load your JSONL file into a Hugging Face Dataset.
  • Each line in data/domain_train.jsonl should be one JSON object, typically like: {"text": "..."}.
  • In Step 6, dataset_text_field="text" tells the trainer to use the text field as the training input.
  • split="train" simply returns a single dataset split. If you want evaluation later, you can add a validation split (for example via train_test_split).