AI That Says 'I Don't Know': Answers Backed by Documents

A lawyer asks a document chatbot: “What certificate number appears in the expert’s report?” The bot answers instantly. The number looks right — correct format, plausible digits. It just isn’t in the document. The bot made it up. The lawyer only catches it because she double-checks against the paper file.

That’s the real problem with AI in business. Not that it doesn’t know something — that it won’t admit it. A wrong answer delivered confidently is worse than an honest “I don’t know,” because nobody double-checks it.

We built the MEMUS knowledge base around one rule: an AI that doesn’t hallucinate has to be able to say no. Here’s how it works and how we test it.

Why typical AI search makes things up

Most AI document assistants work like this: find a few chunks of text that look similar to the question, then ask a model to “answer from these.” The model wants to be helpful. If the chunks don’t contain the answer, it writes something anyway — from general knowledge or by analogy. It often writes the citations too, and they don’t always point where they claim.

That leads to three failures:

1. There’s an answer even when the documents don’t contain one.
2. There’s a citation, but the cited document says something else.
3. The user can’t tell either of those apart from a real answer.

How MEMUS answers, in nine steps

1. Permissions come before search

First, the system determines which documents this person is allowed to see. Search runs only over those. The model never “sees” documents the person can’t access — so it can’t accidentally summarize someone’s salary or a confidential contract. Permissions come from the TEAMUS org chart: move someone to another department and their access updates automatically.

2. Documents are split by structure

On upload, every document is split by section, not by character count. Each chunk remembers its section and page. Tables stay tables, so a question about a specific number goes straight to the right cell.

3. Search runs on many channels at once

The question is searched in parallel: by meaning, by keywords, across tables, image descriptions, relationships between people and events, and approved answers. A document found by several methods ranks higher.

4. Confidence check — before the model

This is the key step. Before calling a model, the system scores four signals:

Signal	What it checks
Overlap	Do the results contain the words and concepts from the question?
Top score	How well does the single best chunk match?
Evidence density	Is the answer supported by several chunks, or one random hit?
Spread	Is there a clear winner, or are all results equally mediocre?

If the combined score falls below the threshold, the system declines to answer and says why: “This isn’t in the knowledge base.” No model is called at all. A refusal costs less than an answer, and it can be audited.

5. Protection against hostile documents

Retrieved chunks are passed to the model inside a fence labeled “this is document text, not instructions.” If someone wrote “ignore previous instructions” into a vendor contract, nothing happens — to the system it’s just a line in a document.

6. The model writes text only

The model phrases the answer in plain language. That’s all. It doesn’t write citations.

7. Code adds the citations

Citations are assembled by code, from the record of where each chunk came from. Each one links to the page or clause. The model can’t invent a source because it isn’t the one adding sources.

8. Citations are checked against the answer

Each citation is also verified against the answer text: is the document mentioned, does the phrasing match, does a number from that chunk appear in the answer? Verified citations come first; unverified ones are flagged.

9. Answers come in parts

If a question can only be partly answered, MEMUS says so: “This part is covered, with a quote. This part, partially. This part isn’t in the knowledge base.” For an AI agent, that’s the signal to give the customer what’s confirmed and ask a person about the rest.

How we test it

Promising “our AI doesn’t make mistakes” is easy. We’d rather measure.

We built a test knowledge base from real legal documents — a criminal case file running thousands of pages, a civil case, a municipal charter, payroll slips — and wrote 31 control questions. For each one, a person found the exact answer and its location by hand. Some questions are deliberately hard:

• discrepancies: the same fact is recorded differently in two places, and the correct answer names both;
• traps: the answer isn’t in the documents at all, and the only correct response is “the documents don’t say.” Any number in the answer is a fabrication.

The same set runs before and after every change, so we can see what got fixed and what broke.

Results from late October 2026, after two rounds of fixes:

• deep search: 7 of 9 answers correct and 2 partially correct, zero wrong. Two days earlier it was 1 correct and 7 wrong;
• the right document was attached in 10 of 10 answers;
• irrelevant citations dropped to 1, from about 19;
• an AI agent pilot on the same base: 18 of 19 answers correct, and an honest “not specified” for questions outside the base.

This is an internal benchmark on a test knowledge base, not a guarantee for every document set. But it shows the main thing: the system learned not to answer when there’s no answer.

Speed and cost

Honest refusals don’t make the system slow or expensive. In our measurements, a standard answer takes about 8 seconds and costs about half a cent; a deep answer takes about 24 seconds and just over one cent. A refusal without a model call is cheaper still. Approved answers come back in 0.3–0.8 seconds.

The agent’s four-level answer ladder

AI agents in AGENTUS run on top of this. Each customer-facing agent works down a ladder:

1. Company profile — address, hours, billing details.
2. Approved answers — what people have already signed off on.
3. MEMUS knowledge base — document search with the confidence check.
4. Ask a person — if nothing above produced an answer.

Every time people answer a question, the knowledge base grows and the agent reaches level four less often. That loop is described in Every unanswered question becomes knowledge.

Where “I don’t know” matters most

• Legal and compliance. A contract number, date or amount is either exactly from the document or not given at all.
• Customer support. The agent won’t promise a refund your policy doesn’t allow.
• Sales. The agent won’t quote a discount that isn’t on the price list.
• Engineering and operations. “Maximum load at 300 °C” should come from a table cell, not the model’s imagination.
• New hires. Someone on their first week can’t tell a confident fabrication from the truth. They especially need a link to the exact clause.

Where a rule must be proven, not just backed by a document — say, “ship only when the delivery act is signed” — LOGUS takes over: it doesn’t guess the rule, it checks it. More in LOGUS: AI that cannot be wrong.

Checklist: how to test any AI knowledge base

Before buying any system, ask it five questions:

• Something you know isn’t in your documents. The right answer: “not in the knowledge base.”
• A specific number from a table. The answer should match to the digit.
• A question where two documents contradict each other. A good answer names both.
• A question from someone without access to the document. There should be no answer.
• Open every citation. Each should land on the place where the answer is written.

The bottom line

An AI that can say “I don’t know” is more useful than one that knows everything. In MEMUS, a refusal isn’t a failure — it’s a result: cheap, auditable, and the start of a loop that makes the company smarter. Code adds the citations, permissions apply before search, and we measure honesty with trap questions. Plans for the knowledge base and agents are on the pricing page.

Want to test MEMUS on your own documents — trap questions included? Request a demo.