Fine-Tuning Dataset Preparation & Formatting

+---------------------------------------------------------------------------------+
|                        DATASET CURATION BEST PRACTICES                          |
+---------------------------------------------------------------------------------+
|  Quality > Quantity: 1,000 pristine examples beat 50,000 noisy scraped rows!    |
|  Diversity: Ensure wide variety of schemas, edge cases, and error handlings     |
|  Format: ChatML format (<|im_start|>system...<|im_start|>user...<|im_start|>)   |
+---------------------------------------------------------------------------------+

Dataset Format: Standard ChatML JSONL

The industry standard for instruction fine-tuning is messages format in JSON Lines (.jsonl), which maps directly to OpenAI-compatible chat templates:

{"messages": [{"role": "system", "content": "You are a PySpark migration expert."}, {"role": "user", "content": "Convert this pandas code to PySpark: df['age_next_year'] = df['age'] + 1"}, {"role": "assistant", "content": "from pyspark.sql import functions as F
df = df.withColumn('age_next_year', F.col('age') + 1)"}]}
{"messages": [{"role": "system", "content": "You are a PySpark migration expert."}, {"role": "user", "content": "Convert this pandas code to PySpark: df_filtered = df[df['status'] == 'ACTIVE']"}, {"role": "assistant", "content": "import pyspark.sql.functions as F
df_filtered = df.filter(F.col('status') == 'ACTIVE')"}]}

Automated Dataset Validator Script

Before starting an expensive GPU training run, validate your dataset for schema errors, empty fields, and token length distribution:

import json
from transformers import AutoTokenizer

def validate_fine_tuning_dataset(jsonl_path: str, model_id: str = "meta-llama/Meta-Llama-3-8B-Instruct"):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    lengths = []

    with open(jsonl_path, "r", encoding="utf-8") as f:
        for idx, line in enumerate(f):
            try:
                record = json.loads(line)
            except Exception as e:
                raise ValueError(f"Line {idx} invalid JSON: {e}")

            assert "messages" in record, f"Line {idx} missing 'messages' key"
            messages = record["messages"]
            assert len(messages) >= 2, f"Line {idx} must have at least user and assistant message"
            assert messages[-1]["role"] == "assistant", f"Line {idx} last message must be assistant"

            # Format using tokenizer chat template
            formatted_text = tokenizer.apply_chat_template(messages, tokenize=False)
            token_count = len(tokenizer.encode(formatted_text))
            lengths.append(token_count)

    print(f"Validated {len(lengths)} examples successfully!")
    print(f"Token Lengths: Min={min(lengths)}, Max={max(lengths)}, Avg={sum(lengths)/len(lengths):.1f}")

# Example check
# validate fine tuning dataset("pyspark dataset.jsonl")