LLM Agents and Finance Applications

Where this lecture is going

Why a brilliant model that just sits there isn't enough

A plain language model turns a prompt into words and stops. It has no memory between calls and can't reach out into the world — so real analyst work breaks it.

• Documents don't fit — a call plus a 10-K can overflow what the model can read at once (frontier-class: 128,000+ tokens ≈ 100 pages)
• Cross-referencing two sources is unreliable when both are crammed into one prompt
• The follow-up restarts from scratch — "now compare to Microsoft" forgets everything

An agent is the same model, wrapped in a loop

Take that one-shot oracle and put it inside a cycle: look at the situation, decide what to do, do it, read the result, repeat. That loop is what makes the LLM an autonomous agent.

Perceive, reason, act — then do it again

Strip the jargon and an agent repeats three moves: take in new information, decide on an action, and let the world respond. The diagram below shows the cycle.

What each piece of the loop means in an LLM agent

Policy, memory, action, and observation each map directly onto things you can see and control in code.

Think out loud, then act — and the act grounds the thought

ReAct (Yao et al., 2022) interleaves a written thought with a real action. The model says what it intends, takes a step, reads the result, and continues from facts rather than guesses.

Three tool calls replace one risky guess

Query: "What was Tesla's stated liquidity-risk policy in its most recent 10-K?" Watch the agent walk to the answer instead of inventing it.

Agents that explore options and learn from their mistakes

Two upgrades make agents smarter without retraining: search several plans before committing, and write yourself a note about what went wrong.

What to hold in your head vs. what to look up

Agents use three kinds of memory, just like a person: what's in front of you now, a library you search by meaning, and a filing cabinet you look up by exact key.

• In-context — the current conversation and recent tool results; instant, but gone at session end
• Retrieval-augmented — a vector database searched by meaning; good for semantic queries over 10,000 SEC filings
• Long-term — a relational or graph DB for exact lookups: entity metadata, fiscal calendar, analyst-revision history

A bad retrieval poisons everything downstream

When an agent searches its library by meaning, it can miss the right document or grab the wrong one — and the reasoning built on top inherits the error.

A tool is a labelled action with a strict contract

Each tool is just three things: a name, a contract for its inputs and outputs, and the code that runs. The contract is what lets the model use it without guessing.

The model asks; a trusted environment acts

Crucially, the model never runs code itself. It emits a structured request, and a separate trusted environment executes it and hands back the result — which is what makes tool use auditable and safe to restrict.

1. Tools are declared in the system prompt as a JSON array
2. The model generates a structured tool-call message — function name + arguments
3. The execution environment dispatches the function and returns the result as a new message
4. The model continues reasoning with that result now in context
5. If the JSON is malformed, a validation error is returned as feedback — the agent regenerates with corrected arguments (Pydantic is widely used for this)

Toolformer and Gorilla: teaching the model when and which tool to call

Two key systems solved the hardest parts of tool use: knowing when a tool call is needed, and selecting the right one from thousands of options.

Wrapping a DCF so the model can't misuse it

Define a tool with typed, described, range-checked inputs. The descriptions teach the model how to call it; the validation stops bad calls before they run.

from pydantic import BaseModel, Field

class DCFInput(BaseModel):
    free_cash_flows: list[float] = Field(
        description="Projected FCFs in USD millions, chronological"
    )
    wacc: float = Field(
        description="WACC as a decimal (0.08 = 8%)", ge=0.0, le=1.0
    )
    terminal_growth_rate: float = Field(
        description="Perpetual growth rate as a decimal", ge=0.0, le=0.15
    )

• Pydantic field descriptions propagate into the JSON Schema the model receives
• Validation errors come back as structured feedback — the agent regenerates with corrected arguments
• Safer than raw SQL: the action space is restricted to meaningful financial operations

An error is an observation, not a dead end

Real tools time out, hit rate limits, and return nothing. A production agent treats each failure as information and recovers in layers.

1. Innermost — retry with exponential back-off for transient failures (network timeouts, rate limiting)
2. Middle — read the error as an observation and try a different action: another endpoint, a relaxed date range, a decomposed query
3. Outermost — call RequestHumanHelp(description) rather than returning a made-up answer

Skills package the work; hooks watch it happen

Two abstractions keep agent systems manageable: a skill is a reusable sub-workflow, and a hook is a callback that fires on events without touching the core reasoning.

Agents, skills, and hooks as plain text — readable, diffable, auditable

Agentic coding assistants (like Claude Code) realise agents, skills, and hooks as plain-text markdown files under version control — every capability and guardrail is a tracked file a reviewer can read.

Where agents, skills, tools, and hooks actually live

Both Claude Code and Cline read plain files from your repo. The four capabilities map to concrete locations — and both ship with the same standard built-in tools you never have to define.

An audit log and a safety-net commit — without touching the agent

In Claude Code a hook is a shell script wired to a lifecycle event in settings.json. Two we'll add today turn good-practice habits into automatic guarantees.

A team of specialists beats one generalist

Split a hard task across narrow specialists and route between them. Each agent does one thing well, and the decomposition itself reduces errors.

Don't memorise the library — look things up at query time

A large asset manager's research library has 50,000 documents. No context window holds it and no fine-tuning reliably memorises it. So retrieve the handful of relevant passages when the question arrives, and reason over those.

Semantic search at scale — finding meaning, not just keywords

A vector database stores document embeddings and answers "which documents have a similar meaning to this query?" in milliseconds — even over millions of documents.

The cheap setup that wins — plus the filter that saves it

Most production RAG is deliberately simple: a frozen off-the-shelf retriever feeding a closed-source LLM through an API. The big reliability win is filtering by metadata first.

In finance, faithfulness comes before everything

The RAGAS framework (Es et al., 2023) scores a RAG pipeline on four axes. For regulated finance, they are not equally important — an unfaithful answer is a hallucinated claim presented to a client.

1. Faithfulness first. Unfaithful = hallucinated = a compliance failure with clients. Below 0.85 → block automated publication, trigger human review.
2. Context recall next. High faithfulness with low recall = accurate but incomplete — material disclosures may be omitted.

Do agents even need a vector database? Often not.

Classic RAG builds an index before any question is asked. Frontier agentic systems — Claude Code, Cline — increasingly skip it: they retrieve just in time with the tools they already have, reading full-fidelity source instead of pre-chunked embeddings.

Turning a call transcript into a structured, cited brief

An earnings-call agent runs a five-stage pipeline — from raw transcript to a structured summary where every claim cites the transcript. Total pipeline latency: 3.2 minutes for a 90-minute call.

1. Ingestion — transcript from a vendor API (Refinitiv, S&P Global) or EDGAR
2. Segmentation — classify speaker turns: management guidance / analyst Q / clarification
3. Metric extraction — find numerical guidance; compare to prior guidance and consensus
4. Sentiment — a directional score per segment on a financially calibrated scale
5. Synthesis — a structured summary with transcript citations (including timestamp and speaker)

Describe verified data — never write from memory

The core principle of automated research: a data agent assembles verified numbers first, then a writer agent describes that package, citing every figure. Free-form "write a report about NVIDIA" produces hallucinated figures.

Natural-language analytics for the portfolio manager

A portfolio research assistant takes natural-language questions and answers them with precise computation — no bespoke code needed for each query.

From Tetlock to GPT: richer, more nuanced signals than bag-of-words

LLM agents extend the long tradition of text-based signals by producing structured outputs — sector sentiment, management confidence, guidance revision direction, novelty relative to prior disclosures — rather than a single number.

Latency, cost, and reliability — the three constraints that matter in finance

A research prototype that takes two minutes to answer a question is fine. The same latency in a live trading workflow is not. Production deployment imposes three hard constraints simultaneously.

Match the level of human oversight to the stakes

Not every action needs a human signature. Pick the oversight pattern by how reversible and how consequential the action is. Ouyang et al. (2022) showed that human feedback is essential for aligning LLM outputs — in finance this applies to factual correctness, not just values.

• Irreversible + high-consequence → pre-execution approval
• Reversible + medium-consequence → checkpoint review
• Low-stakes + high-volume → post-hoc audit

A tamper-evident log of what the agent did — and why

Every action is recorded with a timestamp and full payload, and the records are chained so that altering any past entry breaks every later hash. The agent's own thoughts double as its explanation.

When a document tells your agent to misbehave

Prompt injection is malicious instructions hidden in tool outputs or retrieved documents that try to override the system prompt. Worse, the attacker need not touch your system at all (Perez et al., 2022; Greshake et al., 2023).

Defence in depth — no single control is sufficient

1. Input sanitisation — strip instruction-like patterns from retrieved text
2. Privilege separation — trusted system prompt vs. untrusted retrieved docs processed in a sandboxed context
3. Invariant monitoring — a separate agent blocks actions that violate invariants ("never recommend companies not on the approved list")
4. Dual-key authorisation — consequential actions need human + agent confirmation; capability-based security for API keys

AI governance for finance is arriving — build for it now

In 2023, a New York attorney was sanctioned for submitting a legal brief citing six non-existent cases generated by ChatGPT (Mata v. Avianca). In finance, the stakes are at least as high — and regulators are responding.

The agent stack — and where Lecture 5 goes

Pick the framework that fits the task and the audit trail

Three popular orchestration frameworks make different bets. For regulated finance, the deciding factor is often who logs the full conversation by default.

SPLADE: the best of both worlds — semantic meaning, sparse efficiency

Standard hybrid search combines a fixed dense embedder and fixed BM25. SPLADE goes further: it learns a sparse representation that captures meaning rather than just exact terms.

Meaning-search and keyword-search cover each other's gaps

Dense retrieval captures semantic similarity but fails on exact matches like CUSIP numbers or regulatory citations. Sparse keyword search nails those. Blend the two.

• Higher α (favour dense) — QA over long financial documents; start around α ≈ 0.7
• Lower α (favour sparse) — keyword-heavy regulatory lookup; start around α ≈ 0.3
• Reciprocal Rank Fusion (RRF) — avoids normalisation entirely: combines rank positions from each retrieval system with constant k = 60
• Tune on a held-out validation set — the optimal α is task-specific

How you slice a 200-page filing decides what you can find

A 10-K has hierarchical structure. Cutting it into fixed-size blocks severs meaning at arbitrary points — better to respect the document's own boundaries.

1. Fixed-size — L tokens with δ overlap; simple, ignores structure
2. Semantic — sentence/paragraph boundaries; EDGAR item structure for SEC filings provides natural top-level sections
3. Hierarchical — a multi-level index (document → section → paragraph); retrieve the section first, then the paragraph within it. Two-stage approach reduces context sent to the generator while preserving structure