In short: An AI assistant that answers from company documents uses a technique called RAG: the documents are split into chunks, the system finds the ones relevant to the question and the model answers using only those, citing its sources. Quality depends on tidy documents, respected permissions, citations, an ‘I don't know’ answer and testing with real questions.
In many companies the information exists but nobody can find it: procedures in PDFs, contracts in folders, product manuals, old emails containing the right answer. An artificial intelligence assistant can read these documents and answer staff questions in their own language. Provided it is built properly.
How it works, in plain terms
A language model on its own answers from memory and sometimes makes things up. To make it answer from your documents, a technique called RAG (retrieval-augmented generation) is used:
- The documents are read and split into chunks.
- When someone asks a question, the system looks for the most relevant chunks.
- It passes the question to the model together with those chunks.
- The model writes the answer using only that information and shows which documents it came from.
The model is not ‘retrained’ on your data: it consults the data at the moment the question is asked. So if you update a document, the answer changes.
What it's really useful for
- Internal support: ‘how do I log an international return?’.
- Sales: finding a product's specifications or the terms of a standard contract.
- New starters: getting the procedures in a few minutes instead of asking a colleague.
- Professional firms: searching case files, opinions and historical records.
What determines quality
The documents
An assistant repeats what it finds. If the procedures contradict each other or are five years out of date, its answers will be too. Before you start, you need order and someone responsible for each group of documents.
Permissions
Not everyone should see everything. The assistant must respect the same permissions as the documents: someone who can't open a contract must not be able to read it through the assistant.
Cited sources
Every answer should say which document and which page it comes from. The reader can check in one click. Without citations you can't trust it.
The ‘I don't know’ answer
A good assistant admits when it can't find the information. If it always gives an answer, there is a risk it is making things up. This is set up precisely and tested with trick questions.
Testing with real questions
Before opening it up to everyone, you prepare a list of fifty or a hundred real questions with the correct answers. You measure how many it gets right and where it goes wrong. You repeat the test after every change.
Where the data lives
This is where the level of risk is decided. There are three routes:
- A commercial model through an external service, with sensitive data filtered out before sending. Simple and powerful, suited to documents that aren't very confidential.
- An open model on a dedicated rented server in a European data centre. The documents don't leave your environment.
- A server on your premises for those who want everything to stay physically inside the company.
For contracts, customer data or medical records, the last two are worth considering. We compare them in private AI: rented server or on your premises.
What not to expect
- It doesn't replace the people who make decisions. It provides information that needs checking.
- It can't make up for muddled documents.
- It isn't ‘plug in and forget’: documents change and the assistant needs to be kept up to date and checked.
Where to start
Pick a single group of documents and a small team of people. For example, the customer service procedures and the people who handle customer service. Prepare the test questions. Measure the time saved over a month and decide whether to extend it.
Further reading: AI assistants for your documents · AI for business · rules for using AI at work
Frequently asked questions
What is RAG?
Retrieval-augmented generation: the system retrieves the relevant passages from your documents and passes them to the model along with the question. The model is not retrained: it consults the documents at the moment it answers.
How do you stop the assistant making things up?
By requiring answers based only on the retrieved documents, with the source cited and an ‘I don't know’ answer when the information is missing. This is checked with trick questions.
Where does the data live?
It depends on how confidential it is: a commercial model with filtered data, an open model on a dedicated rented server in Europe, or a server on the company's premises.