The instinct to connect everything
The default plan is to index every document the company owns and let retrieval sort it out. This is expensive, slow to build, and produces worse answers than a narrow corpus — because irrelevant, outdated, and contradictory material competes with the right source at retrieval time.
More context is not more accuracy. Past a point it is noise with a higher bill attached.
Scope by question, not by system
Start from the twenty questions the feature must answer and work backwards to the smallest set of documents that answers them. Most support assistants need a few dozen articles and recent ticket history, not the entire wiki and eight years of email.
That exercise usually reveals that the useful corpus is a fraction of the available one, and that a meaningful part of it does not exist yet in any written form.
Freshness beats volume
A small, current corpus outperforms a large, stale one on nearly every question that matters, because most questions are about how things work now. Prioritise a reliable update path over breadth of coverage in the first version.
A model quoting last year’s policy confidently is worse than a model saying it doesn’t know.
Kill the contradictions before you index
Where two documents disagree, retrieval will surface whichever is more textually similar to the question, which is effectively random with respect to correctness. Resolving contradictions is unglamorous data work that improves output quality more than any prompt change.
Treat it as a permanent function rather than a migration task. Documentation drifts back into disagreement the moment people start editing it again.
Decide what must never be retrievable
Access control has to be part of retrieval, not a layer above it. Internal notes, other customers’ records, pricing exceptions, and anything covered by a contract need to be excluded at the index level, because a model asked nicely enough will otherwise repeat whatever it can reach.
This is the requirement most likely to be discovered late, and the most expensive to retrofit. Ask the question in week one.