Large Language Models: Architecture and Practice

Where this lecture is going

How do you turn a 100-page filing into one vector?

Earnings calls run about 10,000 words; 10-K filings about 100,000. Three ways to compress all those word-vectors into a single document representation — each with a catch.

One word, many meanings — static embeddings pick one and ignore the rest

Word2Vec and GloVe assign a single fixed vector to each word. In finance, polysemy — one word, multiple meanings — is everywhere and consequential.

• bank — financial institution OR a river's edge; a sentiment model trained on finance incorrectly transfers positive sentiment to sentences about river flooding
• yield — bond yield, crop yield, or to give way
• default — failure to repay a debt OR a software setting
• position — a long/short holding OR a physical location

RNNs and LSTMs: what they got right, and why they lost

For a decade, recurrent networks (RNN → LSTM → GRU → BiLSTM) were the default for financial NLP — named-entity recognition, sentiment, event extraction. They read a sequence one token at a time, carrying a running "memory" forward. Three structural limits are exactly what the Transformer was built to remove.

What if the model could look at every word at once?

The bottleneck in older translation models: the encoder had to squeeze the whole input into one fixed vector. Bahdanau et al. (2015) let the decoder instead look back at every input word and weight the ones it needs right now.

Every word asks every other word: "are you relevant to me?"

Each word produces three roles — a query (what I'm looking for), a key (what I offer), and a value (what I'd contribute). Matching queries to keys decides how much of each value to pull in.

Run attention several times in parallel — each head a different lens

One attention pattern can only capture one kind of relationship. Multi-head attention runs several in parallel, then concatenates them — so different heads can track grammar, co-reference, and figures at once.

What do heads learn? (Clark et al., 2019)

• Lower layers — syntax: subject–verb agreement, negation scope
• Deeper layers — semantics: entity co-reference, temporal relations
• Financial models — heads specialise on "last quarter" → figures, entity linking

Self-attention is order-blind — so position is injected explicitly

Self-attention treats the input as a set: shuffle the words and the math is unchanged. A sinusoidal positional encoding tags each token with its location so the model can recover order.

• Low-indexed dimensions oscillate rapidly (short period) — they pin down fine position within a few tokens.
• High-indexed dimensions change slowly (long period) — they encode coarse position across the whole sequence.
• Stacking the two gives every position a unique signature — a token at position 5 never collides with one at position 50.
• Any fixed offset \(\Delta\) is a linear transformation of the encoding, so relative positions are linearly decodable: the model can learn "\(k\) tokens apart" directly.

Every LLM reserves a few tokens for structure, not words

Alongside word-pieces, LLMs keep a handful of tokens that carry structural rather than lexical meaning — and getting them wrong breaks financial pipelines in subtle ways.

One Transformer block: attend, then think, with safety rails

A block does two things — mix information across words (attention), then process each word on its own (a small feed-forward network) — each wrapped in a "shortcut" that keeps training stable.

Seven steps from tokens to next-word probability — in miniature

A toy GPT (6-word vocab, \(d_\text{model}=4\), one decoder block) reads "revenue fell" and predicts the next word. The same seven steps run inside every production LLM — just with larger matrices.

1. Embed — look up each token's row in the embedding table \(E\)
2. Add positional encoding — inject sinusoidal signals so the model recovers order
3. Form Q, K, V — project each row to a query, key, and value vector
4. Score, mask, softmax — dot-product scores, causal \(-\infty\) mask, row-wise softmax → attention weights \(A\)
5. Attention output + Add & Norm — weighted average of values, then residual + LayerNorm
6. Feed-forward + Add & Norm — position-wise FFN (widen → ReLU → project), then residual + LayerNorm
7. Un-embed + softmax — multiply final row by \(E^\top\) to get logits; softmax gives \(P(\text{next token})\)

Step 1 — "Embedding lookup" is literally one matrix times another

Our toy model knows a vocabulary of just \(V=6\) tokens — every "word" it can read or write:

Each token owns exactly one row of the embedding table \(E\ (6\times4)\) — so \(V=6\) rows, one \(4\)-dimensional vector apiece. A production GPT swaps this \(6\) for a byte-pair vocabulary of ~100,000 tokens, but the mechanism below is identical.

The prefix "<s> revenue fell" has token ids \((0,1,2)\). Stack their one-hot rows into \(O\) and multiply by the embedding table \(E\) — each one-hot row just selects a row of \(E\):

Step 2 — Add position, so order is not lost

Self-attention is permutation-equivariant — on its own it cannot tell "revenue fell" from "fell revenue". We add a sinusoidal positional encoding \(\mathrm{PE}\) (\(n=3,\ d_\text{model}=4\)) to the embeddings:

Row \(t\) of \(X^{(0)}\) now encodes both the word and where it sits. From here the model works only with these vectors.

Step 3 — Project each row into a query, key, and value

Each token row is multiplied by three weight matrices. To keep the arithmetic legible we set \(W^Q=W^K=I\) (so \(Q=K=X^{(0)}\)); only \(W^V\) mixes the coordinates:

\(Q\) asks "what am I looking for?", \(K\) advertises "what do I offer?", \(V\) carries "what I'll pass on if attended to."

Step 4 — Scores → causal mask → softmax = attention

Score every pair with \(S=QK^{\top}/\sqrt{d_k}\) (\(\sqrt{d_k}=2\)); the causal mask sets the future to \(-\infty\) so a token cannot peek ahead; a row-wise softmax turns each row into weights \(A\):

<s> can only see itself; fell (row 3) spreads its attention \(0.13/0.17/0.70\) — weighting its own position most. Every row of \(A\) sums to \(1\).

Step 5 — Attention output, then Add & Norm

Each token's new vector is the attention-weighted average of the value rows, \(\mathrm{Att}=AV\). Add the residual input and layer-normalise: \(Z=\mathrm{LayerNorm}\!\left(X^{(0)}+\mathrm{Att}\right)\):

LayerNorm acts on each row (zero mean, unit variance) — which is why every row of \(Z\) sums to zero. The residual keeps the original signal in play.

Step 6a — Feed-forward: widen to 6 dims, then ReLU

The position-wise FFN, \(\mathrm{FFN}(\mathbf{z})=\max(0,\mathbf{z}W_1)W_2\), runs on each row independently. First widen \(4\to6\) with \(W_1\) and clip negatives with ReLU:

Hidden units 4 and 6 are negative at every position, so ReLU switches them off (the \(0\) columns) — a concrete instance of the sparsity ReLU induces.

Step 6b — Project back to 4 dims, then Add & Norm

Project the 6-dim hidden layer back down with \(W_2\), then a second residual-and-norm gives the block output \(X^{(1)}\):

\(X^{(1)}\) is the output of one decoder block. A real GPT stacks 32–96 of these — each the same two sub-layers traced here.

Step 7 — Un-embed → logits → softmax: the next word

With tied weights the logits are \(\mathrm{logits}=X^{(1)}E^{\top}\). Only the last row matters for generation — it scores every vocabulary word as the continuation of fell:

The last row's top score is column 4, sharply — the dot product of \(X^{(1)}_3=(-0.95,1.00,1.00,-1.05)\) with sharply's embedding \((0,1,1,0)\) is \(1.00+1.00=2.00\). A row-wise softmax turns it into probabilities:

BERT learns by playing fill-in-the-blank

Hide 15% of the words in a sentence and train the model to guess them from the words on both sides. Reading in both directions makes it a powerful reader — but not a writer.

GPT learns by always guessing the next word

Read left to right and predict what comes next, over and over. That single skill, at scale, becomes fluent generation — and it's why one architecture can draft, answer, and reason.

Which architecture for which job?

Chain-of-thought + RL reward: the next frontier beyond standard generation

Standard LLMs generate tokens one step at a time with no backtracking. Reasoning models are trained to produce an explicit thinking trace before committing to an answer — dramatically improving multi-step financial tasks.

• Standard generation — one autoregressive pass; fast but commits to the most plausible next token without deliberation
• Chain-of-thought prompting (Wei et al., 2022) — instruct the model to show its steps; each step independently verifiable
• Reasoning models (o1, o3, DeepSeek-R1) — trained via RL with verifiable rewards to generate a latent trace \(\mathbf{r}\) before producing answer \(\mathbf{a}\)

Instruction tuning + RLHF: teaching a model to follow orders

A raw pre-trained model completes text; it doesn't follow instructions. Two extra training stages turn it into a reliable assistant that respects structured output formats and financial conventions.

Who builds the models you'll actually call?

A handful of general frontier models plus a few finance-specific ones. Context length (how much text it can read at once) and open vs. closed weights matter as much as raw size.

• FinBERT: two independently trained models share the name — Araci (2019) on Reuters/Bloomberg/10-K; Huang (2023) on analyst-labelled sentences. Do not conflate in benchmarks.
• BloombergGPT (Wu et al., 2023): 363B finance + 345B general tokens. Outperforms same-size general models on FinancialPhraseBank and ConvFinQA. Large frontier models often match it via scale.
• Open-weight models (Llama family) enable on-premises deployment — no data-privacy risk. 405B at 16-bit requires ~810 GB GPU memory.

One knob decides whether the model plays it safe or improvises

At each step the model has a probability over possible next words. Temperature, nucleus, top-\(k\), and beam search are different rules for picking from that distribution.

• Temperature — low = predictable, high = adventurous
• Top-\(k\) — only sample from the \(k\) highest-probability words
• Nucleus (top-\(p\)) — only sample from the smallest set of words covering probability \(p\) (Holtzman et al., 2020); adapts vocabulary size to model confidence
• Beam search — keep the \(B\) best partial sentences; deterministic; prone to degeneration on open-ended tasks

If a machine reads the output, set temperature to zero

The right setting depends entirely on whether you want one correct answer or a range of ideas.

How to make a model return data, not prose

To extract figures into a database you need valid JSON every time. Constrained decoding blocks any token that would break the required schema before the model can pick it.

class EarningsReport(BaseModel):
    revenue_bn: Optional[float] = None
    net_income_bn: Optional[float] = None
    eps: Optional[float] = None

client.messages.create(
  tools=[{"name": "record_earnings",
          "input_schema": EarningsReport.model_json_schema()}],
  tool_choice={"type": "tool", "name": "record_earnings"},
  ...
)

Anthropic tool use + Pydantic: the schema defines exactly what fields come back. tool_choice forces the tool call — no silent fallback to free text.

Three failure modes constrained decoding does NOT fix

Guaranteeing the shape of the answer says nothing about whether the numbers inside it are right.

1. Numerically wrong-but-valid — a clean number that's simply incorrect (pair with \(\tau=0\) + RAG)
2. Silent field omission — model emits explicit null; detectable but requires required/optional field discipline in the schema
3. Unit confusion — millions vs. billions; requires application-level validation with canonical units in the Pydantic model

Models bill by the token — and token math decides your budget

Before processing, text is chopped into sub-word "tokens" (BPE). You pay per token in and per token out, so the same job can cost wildly different amounts across providers.

RAG: don't trust the model's memory — hand it the document

Retrieval-Augmented Generation fetches the most relevant passages first, then asks the model to answer using only those — so every claim traces back to a real source.

Hallucination: fluent, confident, and wrong

A hallucination is output that sounds authoritative but isn't supported by the input or the real world. In finance, three kinds do the most damage.

• numerical D/E ratio of 1.8 reported as 0.8 — indistinguishable from correct figures downstream
• entity wrong company or regulatory body substituted
• citation a fabricated regulation, ruling, or standard (Ji et al., 2023)

No single fix — stack six defenses together

In production you layer complementary techniques; none is sufficient alone.

1. RAG — ground answers in verified passages; model rarely fabricates figures absent from retrieved text
2. Self-consistency sampling — \(K\) chains at \(\tau>0\), majority vote; best for multi-step numerical reasoning (Wang et al., 2022)
3. Chain-of-thought (Wei et al., 2022) — force explicit steps; each independently verifiable; reasoning models implement this natively
4. Claim decomposition (FActScore) — break output into atomic claims; flag unsupported ones before human review
5. Confidence elicitation + abstention — model flags uncertainty; calibrate thresholds against accuracy; confidence-gating for production pipelines
6. Prompted self-critique — a second pass re-checks every claim against the source and retracts unsupported ones; the inference-time analogue of Constitutional AI (Bai et al., 2022), and the one defense a closed-API user can deploy without model weights

The legal and ethical traps practitioners must know

Beyond accuracy, deploying LLMs in finance runs into regulation, contaminated back-tests, privacy law, and built-in geographic bias.

• EU AI Act (Reg. 2024/1689, in force Aug 2024) — credit scoring, insurance pricing, AI-assisted securities pricing are high-risk: conformity assessments, human oversight, transparency to regulators
• Look-ahead bias (Didisheim et al., 2025) — LLMs can reconstruct historical time series from memory, producing spurious back-test predictability; memorised data is contaminated data
• GDPR (EU 2016/679) — fine-tuning or RAG on client personal data triggers right-to-erasure; a model trained on personal data can't satisfy erasure without retraining; data-transfer restrictions on non-EU API routing
• Geographic bias — all five major benchmarks are North American / English; US-trained sentiment models may misread Japanese, Chinese, or Indian financial contexts
• US / fiduciary duty — SEC posture holds the human adviser fully responsible for AI-generated outputs; whether an LLM answer counts as a "recommendation" is unsettled, but "the model said so" is not a defense

A 70B model is too large for real-time trading — compression bridges the gap

A 70B model at 16-bit precision needs ~140 GB of GPU memory and has latency measured in seconds — incompatible with risk monitoring or order routing. Three complementary techniques shrink the model without gutting accuracy.

Trading bits for bytes: deploy a 70B model on two consumer GPUs

Quantisation reduces weight precision from 16-bit float to 4–8-bit integer, cutting memory 2–4× with negligible accuracy loss on most tasks — without any retraining.

What we covered — and where Lecture 3 goes

Why divide attention scores by √dₖ?

Without scaling, dot-product scores grow with dimension, the softmax saturates, and gradients die — reintroducing the very problem attention was meant to solve.

The benchmarks that define financial NLP

FinQA requires multi-step arithmetic over tables — reasoning models dominate. ECTSum evaluates abstractive summarisation against expert-written gold summaries.

Dense, sparse, or hybrid: how RAG finds passages

Dense retrieval matches meaning; sparse (BM25) matches exact strings like CUSIPs and tickers; hybrid fuses both. The combination is the production default.