Grounded LLM generation: keeping the model to the facts

The problem#

Ask a language model to tailor a document to a specific reader (I used résumés against real job postings, because the facts are easy to check) and it will happily do it. It will also, sooner or later, round a number up, add a tool the posting asks for, or claim a result nobody measured. Each of those reads well and is false, and a reviewer who catches one stops trusting the rest.

I wanted generation that is tailored per posting but can only draw on what’s on file, with the checkable parts enforced in code, and a way to measure whether the output is actually good, not just whether it looks good to the person who built it.

Facts in, slots out#

Everything the documents may say lives in a versioned fact vault: every role, bullet, project and skill, in English and French, with a full snapshot kept on every change. The model never writes a document from scratch. It reads a short brief (the posting’s key terms and a menu of vault items) and returns a payload: which items to use, in what order, and optional rewrites of their wording.

Deterministic code then checks the payload and renders it:

• Numbers: a rewrite may not contain a number that isn’t in the original fact, so “cut costs by 30%” can’t appear from nowhere.
• Technologies: a rewrite may not name a tool or technology that appears nowhere in the vault.
• Cross-document consistency: a cover letter that cites a fact the paired résumé left out gets flagged before it goes anywhere.
• Fit: the renderer measures the real PDF and trims the lowest-priority content until it fits the page limit, never dropping a whole role.

A document is a pure function of the template version, the vault version and the payload, so any document can be rebuilt later and checked against a stored hash.

Where the postings come from#

The same system is a Python MCP server that crawls 63,000+ company job boards through 24 applicant-tracking-system adapters into a local database of 1.1M+ postings, normalized into one record shape. Matching and ranking are deterministic keyword scoring plus a small local embedding model. The model makes the judgment calls; the code does everything that can be checked.

Measuring it: agents in the loop#

Unit tests can prove a guard works. They can’t tell you whether a reader finds the output convincing. So the evaluation harness runs the real pipeline end to end: AI agents act as the user on a fixed benchmark of 12 real postings, calling the same tools a person would, 24 runs per round. Blind LLM reviewer panels, briefed as a recruiter, a hiring manager and, for some postings, a senior engineer from the role’s own field, then score the documents against a written rubric, without knowing which version they are reading.

Each round’s findings became fixes: a clearer brief for the model, stricter guards, a better page order. Then the same 12 postings ran again, compared with the documents produced before the work started (panel scores) and the pre-fix pipeline (requirements coverage).

The panels score strictly: on their scale 5 is typical and 7 is shortlisted. The remaining gap is mostly not wording: reviewers asked for outcomes that aren’t on file, and the system won’t invent them.

What I took from it#

• Put the model where judgment is needed (what to emphasize, how to phrase it) and keep everything checkable in code.
• A guard you can’t measure is a guess. The agent-in-the-loop harness turned “this looks better” into numbers I could compare across rounds.
• Most of the fixes were better inputs (a rebuilt fact vault) and clearer instructions to the model.

Stack: Python, MCP, SQLite, a small local embedding model, HTML-to-PDF rendering, and 1,200+ automated tests. The code is private; I’m happy to walk through it. Get in touch.