Bringing an LLM Wiki into a Live CRM

Our flagship product is a CRM platform that clients run their daily operations on, with AI agents built into it. Staff ask those agents questions such as "what's the next step on this sales order?" or "the customer's PO price is different from our quote, can I fix it?". A good answer needs two kinds of knowledge: the client's procedures (SOPs and business rules) and their live records (companies, contacts, orders, invoices, projects).

The knowledge base behind it is an LLM wiki, an approach a coworker first set up at the company, inspired by Andrej Karpathy's idea. Instead of chopping documents into vectors, an LLM reads the raw sources and writes them up as tidy, human-readable pages, and the AI answers from those pages. My task was to apply that structure to the flagship product, which turned out to be a different problem from the one the wiki was first built for.

The Problem

A wrong answer needs a traceable cause

With company SOPs, a wrong answer can come from two very different places:

1. The rule itself is wrong or out of date. The fix is to update the SOP.
2. The rule is right, but the model ignored it. The fix is in the agent: the prompt, the search, or the model.

From the outside, these look exactly the same. We needed a way to tell them apart every time.

Vectors hide the source

The common approach, RAG (retrieval-augmented generation), splits documents into chunks and turns each chunk into a vector. It works, but once the text is a vector you can't read it anymore. When an answer goes wrong, it's hard to say which text it came from, or whether that text was even the right one.

The wiki was built for documents, not live data

The original wiki was designed for handbooks, policies and specs. They change slowly, and an LLM rewriting them into pages is exactly the point. A CRM holds thousands of records that change every day, and the numbers in them have to be exact. Letting an LLM "summarise" a price or a quantity wasn't acceptable.

Requirements we had to meet

• Every answer cites the pages it came from, and those pages trace back to the original source.
• Records are copied exactly, with no LLM in between.
• The wiki stays in sync with the CRM as records are created, changed and deleted.
• Procedures and records live in one search without crowding each other out.

The Solution

Why a wiki over RAG

In an LLM wiki the source stays intact. Every page is plain text that a person can open and read, says which source it came from, and has a stable id. When the AI answers from a page, we can follow the answer back to that page, and from the page back to the original document or record. That traceability was the reason the company chose this approach.

Two lanes instead of one

I split ingestion into two lanes:

• Records go through a deterministic lane, with no LLM. A CRM record is already structured, so there's nothing to interpret. Each record becomes one page through a fixed template, which makes it fast and repeatable.
• Everything else goes through the LLM lane. SOPs, specs and files attached to records (including scanned documents, through OCR) are still read and written up by an LLM, the way the wiki was designed.

Each page has exactly one writer. A record never gets a second, lossy copy from the LLM lane.

Every page points back to the system of record

Each record page cites the exact CRM row it came from, and carries the time that row last changed. A file attached to a record points to its parent record, so even a scanned PDF can be traced back to where it came from.

Pages also leave out the things that move. A sales order's page holds the details that stay put, not its current status. When the agent needs a volatile field, it checks the live CRM. If the page and the live value disagree, the live value wins, and the answer says so.

Keeping up with live data

The CRM sends a webhook when a record changes, a scheduled sweep fetches what changed, and the changes are written into the wiki in batches. Deleted records are retired from the wiki instead of lingering as stale pages.

Citations as part of the answer

The product's existing knowledge search tool calls the wiki's retrieval endpoint, so every agent on the platform gets the wiki through one tool, with the product's own permissions. Each result comes back with its page id as the citation.

Each answer carries the pages it retrieved and marks the ones it actually used. The model can only cite a page the search really returned, so it can't invent a source. When someone flags a bad answer, we can open the page the agent relied on and check two things: did the page say the right thing, and did the answer follow it?

Search tuned for procedures and records side by side

A record page shares most of its words with the procedure about it: "supplier payment slip" appears on the SOP and on every purchase order that has one. So I added:

• A collection filter, so a procedure question searches only the procedure pages.
• Title-first ranking, so searching for a record number returns that record before the documents that merely mention it.
• Matching sections for long pages, so the agent gets the part of the page that answers the question, not just its opening.
• Small query fixes: trimming punctuation, dropping filler words, and matching singular and plural.

Key Outcomes

• Traceable answers. Every answer links to the pages it used, and every page links back to its document or CRM row.
• Wrong rule vs wrong model, told apart. A flagged answer shows what the agent read and what it relied on.
• Exact records. CRM data reaches the wiki without an LLM rewriting it.
• A wiki that keeps up. Changes and deletions in the CRM flow into the wiki on their own.
• One search for both kinds of knowledge. Procedures and records are searched together without crowding each other out.

What Changed

Initial wiki setup	Applied to the CRM
Documents only, rewritten by an LLM	Records copied exactly; documents still go through an LLM
Pages cite a source file	Pages cite the exact CRM row and when it last changed
Ingested on demand	Synced by webhook and a scheduled sweep
One pool of pages	Procedures and records, searchable separately or together
Answers name their sources in prose	Answers carry retrieved and used pages as citations

Trade-offs & Lessons

What worked: Keeping the LLM out of the records lane removed a whole class of wrong numbers. Making each page point at its upstream row meant every citation could be checked against the system of record. And splitting "retrieved" from "used" turned vague complaints about the AI into something we could actually investigate.

Scalability is still the open problem. Search in the wiki is mostly keyword-based, with an embedding-based fallback only when the keywords find too little. That keeps it predictable and explainable, but it has costs as the wiki grows:

• Performance. The search index grows linearly with the number of pages. A few hundred pages is fast. Adding thousands of CRM records as pages makes every index read heavier.
• Accuracy. More pages means more near-matches. Every record that mentions an SOP number is one more page competing with the SOP itself. The collection filter and ranking help, but each fix is a patch for a problem that grows with the data.
• Wording. Keyword search depends on the agent using the words the page actually uses. A full-sentence question can miss a page that a short keyword search finds right away.

What we'd explore next: better ranking, smarter splitting between collections, and a hybrid of keyword and semantic search. Whatever changes, the trade stays the same: I'd rather have a knowledge base I can read and trace, and work on making it scale, than one that scales easily but can't explain a wrong answer.

---

Stack: Python · Cloudflare Workers · Durable Objects · R2 · D1 · NestJS · MCP