Training and Fine-Tuning Large Language Models

What we'll do in the next hour

Four skills, end to end: read a scaling law as a budget, size a LoRA adapter, run a real fine-tune on a finance task, and reason about preference data for alignment.

Three stages from raw text to aligned model

Every modern finance LLM passes through three distinct phases — understanding which phase you are in determines which tools and metrics matter.

LoRA: train a tiny patch, not the whole model

Instead of editing billions of frozen weights, LoRA bolts on two small matrices and only trains those — a cheap, removable "patch."

Chinchilla: model size and data should grow together

For a fixed compute budget there's a sweet spot — roughly 20 tokens of training data per model parameter. Too big a model on too little data wastes the budget.

How many parameters does a LoRA patch actually train?

Model: LLaMA-7B, 32 transformer layers, hidden dimension \(d = 4096\). Each layer has four square projection matrices.

Four questions on the LoRA patch

Using the setup (\(r=4\), Q and V only, 32 layers, \(d=k=4096\)) — work individually or in pairs.

1. A. Compute the total trainable LoRA parameters. Show the step-by-step arithmetic.
2. B. Express this as a fraction of the 7B total. Is it above or below 0.1%?
3. C. Double the rank to \(r = 8\). By what factor does the count change — linear, quadratic, or something else? Justify.
4. D. A colleague prefers older adapter modules. Give one concrete reason LoRA wins for inference-time deployment on a high-throughput trading floor where latency is measured in milliseconds.

Given a compute budget, how big should the model be?

Scenario: an investment bank's AI team has a compute budget of \(C = 10^{22}\) FLOPs to train a financial language model.

Pushing the budget further

1. B. The team instead trains a 1B-parameter model. By the Chinchilla rule, how many tokens for a compute-optimal run? What is the implied FLOP cost?
2. C. A senior researcher argues "train the biggest model possible to maximise accuracy." When is this valid? When is it wrong?

Teach a finance model with a tiny removable patch

Load pre-trained FinBERT and apply LoRA fine-tuning on a handful of labelled financial sentences using HuggingFace PEFT.

Three steps, then three diagnostic questions

Five labelled sentences (positive / negative / neutral)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import LoraConfig, get_peft_model, TaskType

MODEL = "ProsusAI/finbert"
SENTENCES = [
    ("Revenue increased 15% year-over-year.", 0),         # positive
    ("The firm reported a net loss of 200M USD.", 1),     # negative
    ("Operating margins remained broadly stable.", 2),    # neutral
    ("Strong demand drove record quarterly earnings.", 0),# positive
    ("Guidance was withdrawn amid macro uncertainty.", 1),# negative
]

Attach the adapter to query and value only

tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(MODEL, num_labels=3)

lora_cfg = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.05,
    bias="none",
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()  # observe trainable % here

Three epochs, then read the predictions

texts, labels = zip(*SENTENCES)
enc = tokenizer(list(texts), padding=True, truncation=True, return_tensors="pt")
label_tensor = torch.tensor(labels)

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4)
model.train()
for epoch in range(3):
    out = model(**enc, labels=label_tensor)
    out.loss.backward(); optimizer.step(); optimizer.zero_grad()
    print(f"Epoch {epoch+1}: loss={out.loss.item():.4f}")

model.eval()
with torch.no_grad():
    logits = model(**enc).logits
    preds = logits.argmax(-1)
    id2label = {0: "positive", 1: "negative", 2: "neutral"}
    for sent, pred in zip(texts, preds):
        print(f"[{id2label[pred.item()]}] {sent}")

Does the math match what the code reports?

1. Q1 — trainable fraction. Compare the reported trainable count to total FinBERT size (\(\approx 110\)M). How does it line up with the theoretical value for LoRA \(r=8\) on query and value across all 12 BERT layers?
2. Q2 — rank sweep. Change r=8 to r=4 then r=16; rerun the loop each time. Do predictions change? Does loss fall faster at higher rank? What does that say about rank vs. expressiveness on a tiny dataset?

Is patching all four projections worth the extra cost?

Q3. Change target_modules to all four projections: ["query", "key", "value", "dense"]. How does the trainable count change? Do the predictions improve? Is the extra compute justified?

Pick the right tool for each desk

For each scenario, choose: zero-shot / few-shot ICL / LoRA fine-tune / full fine-tune — and say why.

1. Hedge fund. Classify 10-K risk-factor sections "material" vs. "immaterial" for one covenant type. 50 labelled examples, 2-hour deadline, using GPT-5.6 Luna via API.
2. Tier-1 bank. On-premise earnings-call sentiment (no external API), 200,000 docs/month, GDPR data-residency, 20,000 labelled examples.
3. Fintech startup. General-purpose financial chatbot; no task-specific labelled data, but access to a pre-trained instruction-tuned LLM.
4. Quant research. Test whether a 70B model can replicate an expert's reasoning on novel bond-structuring questions. No labelled data exists.

A 7B model fine-tuned by moving ~2 million parameters

The budget says: build small, train fully

Instruction tuning: why format diversity matters

PIXIU / FinMA (Xie et al., 2023) trained on 136,000 financial instruction samples. InvestLM used far fewer but carefully curated samples. Both achieved strong results — for very different reasons.

1. A. The Superficial Alignment Hypothesis says instruction quality matters more than quantity. Design a test: what would you measure on a held-out set to distinguish a quality effect from a quantity effect?
2. B. A financial chatbot needs to handle three phrasings of the same task: "What is the sentiment?", "Classify as positive/negative/neutral:", "Rate the tone of:". How would you structure an instruction-tuning dataset to handle all three without labelling 3× as much data?
3. C. The information-overload effect (Balogh & Didisheim, 2025) shows an inverted-U: accuracy peaks then falls as context length grows. How would you operationalise this finding when designing a RAG pipeline for earnings-call QA?

DPO: aligning a model from preference pairs

You have the LoRA-fine-tuned FinBERT sentiment model from the case study. Now align a small generative finance model with Direct Preference Optimisation on preference pairs \((x, y_w, y_l)\).

1. A. In one sentence: what does raising \(\beta\) do to how far \(\pi_\theta\) may drift from \(\pi_{\mathrm{ref}}\)?
2. B. Why does DPO need a frozen reference policy \(\pi_{\mathrm{ref}}\) at all? What failure occurs if you drop it?
3. C. Construct three plausible \((y_w \succ y_l)\) preference pairs for a financial-advice prompt where "plausible vs. accurate" is the distinguishing axis. Annotate each with the relevant regulatory concern (MiFID II suitability, SR 11-7, or hallucination).