I Let an AI Build My Research Library. Then It Invented a Law That Didn’t Exist.

Andrej Karpathy described a simple, elegant idea: keep your knowledge as plain markdown files, and let an LLM do the grunt work of compiling, cross-referencing, filing and book-keeping. You curate the sources and ask the questions; the model summarises, links, and maintains.

I’m a construction and commercial lawyer. Tinkering with AI isn’t my job — it’s a side project. But you can’t practise in this profession right now and not watch AI arriving in it, and I wanted to actually understand the field rather than skim headlines about it. The trouble is it changes weekly: vendors pivot, valuations move, regulations shift, benchmarks get superseded. A self-maintaining research library was exactly the tool I wanted. So I built one, almost exactly to Karpathy’s pattern.

Drop a source in, say “ingest this,” and a single article would ripple out into five to fifteen interlinked pages. Ask a question, and good answers got filed back as new pages. The knowledge compounded. For a few weeks it felt like magic.

The magic broke the day I actually fact-checked it. The AI-generated synthesis notes I’d been feeding back in — tidy summary memos the model had written from other sources — were quietly seeding errors. A wrong launch date here. A job title that promoted a COO to CEO there. A headline figure no primary source actually supported. Small things. But the wiki had ingested them as fact, cross-linked them, and propagated them across pages. Each copy agreed with every other copy, which made the whole thing look authoritative.

Then came the one that rattled me. An audit found that a synthesis note had fabricated a regulation outright — a specific “technology-competence duty” with a commencement date, a regulator, and disciplinary consequences. None of it existed. The model had fused a few real things — a real money-laundering deadline, a real professional-conduct statement, an American rule with no local equivalent — into a plausible, dated, official-sounding obligation. It had propagated into seventeen pages, worked its way into material I was preparing to share, and survived three previous reviews — precisely because it was consistent everywhere it appeared. Nothing contradicted it, so nothing flagged it.

A knowledge base that compounds is also a knowledge base that compounds its errors. The thing Karpathy’s pattern gave me for free — effortless propagation — was the exact mechanism that turned one hallucination into seventeen.

Here’s what makes this more than a personal anecdote. In March 2026, Anthropic published one of the largest studies ever done on what people actually want — and fear — from AI.

That tracks. The entire value of a lawyer is that you can rely on what they tell you. An AI that is confidently wrong isn’t a productivity tool for that person — it’s a liability with a fast typing speed. The fear is rational, it’s widespread, and in my profession it’s nearly universal.

But here’s the move most of the hand-wringing misses. The reliability problem is not a reason to keep AI at arm’s length. It’s a specification for how to build with it. And — the counter-intuitive part — the most effective tool for fighting AI’s unreliability is AI itself. Two moves do most of the work.

Ground it in data you curated, not the open model. My system’s first rule is that the AI compiles only from a folder of sources I chose, and is forbidden from filling gaps with “general knowledge.” A model answering from a controlled corpus you assembled is a different, far more trustworthy animal than a model free-associating from everything it absorbed in training. The curation is the moat.

Build the verification with AI, too. Every check I’m about to describe — cross-referencing a figure across sources, confirming an executive’s title, testing whether a cited regulation actually exists — is run by the AI, on a leash, against rules I set. I didn’t hire a fact-checking team. I pointed the same technology that writes the draft at the job of attacking it.

The markdown-and-an-LLM skeleton is still Karpathy’s. But almost everything I’ve added since is about one question the original pattern doesn’t address: how do you know which claims to trust?

One more structural rule: every page splits what is (sourced, tagged) from what it means (synthesis). The model compiles the first. The judgement in the second is mine. And a human gate for anything that goes outside — machine-compiled pages stay marked draft until a human flips them to reviewed, and only reviewed pages are cleared for any use beyond my own desk.

While writing this, I had the system chase down a productivity statistic — a “+1.8%” figure I’d seen attributed to a particular AI research paper. The truth-check couldn’t find it in that paper. Under the old regime, I’d have shrugged and dropped the number.

Under “verify the discard,” dropping it wasn’t allowed without proof. So the system went and read the cited paper in full — and discovered two things. First, the figure genuinely wasn’t there; it belonged to a different paper by an overlapping author, which I then tracked down and added properly. Second — and this is the part I’d never have caught otherwise — while reading the paper, it found a separate, pre-existing error sitting in my own wiki: a statistic I’d recorded months ago as “4.5× more likely” when the source actually said “almost fourfold.” I’d misread a percentage (4.5%) as a multiple (4.5×). One had been quietly wrong for months.

Chasing one number I’d nearly discarded surfaced a different error entirely. That’s the whole philosophy in one episode: the discipline isn’t there to make the AI look good. It’s there to make the AI catch itself.

It started as “an LLM that tidies my markdown.” It’s become something closer to a knowledge pipeline with a conscience — an ingestion process with verification gates, an audit trail, a propagation guard, and an explicit, visible epistemics. The AI still does the enormous grunt work Karpathy promised: reading, summarising, cross-linking, filing. I’d never go back to doing that by hand. But the value I’ve actually added on top isn’t more automation. It’s doubt, made systematic.

If there’s a transferable lesson for anyone pointing AI at serious work, it’s this: the hard part was never getting the machine to produce. It was getting it to tell me, reliably and legibly, which of the things it produced I’m allowed to believe. Build that layer first. The output is only as valuable as your ability to trust it — and trust, it turns out, is the part you still have to engineer yourself.