Better retrieval can beat a bigger model
When an internal assistant gives weak answers, the model is not always the problem. The system may be retrieving too much, too little, or the wrong evidence.
An internal assistant gives a weak answer.
The first reaction is often to switch to a larger model.
Sometimes that helps. Often the model received an old document, missed the exact customer record, or had to find one useful paragraph inside a pile of irrelevant text.
The answer cannot be better than the evidence selected for it.
Retrieval is the step before the answer
When an employee asks a company question, the system usually searches approved sources and places selected material into the model’s working context.
That selection is retrieval.
Good retrieval needs to answer several questions:
- Which sources may this employee access?
- Which records match the customer, project, or policy?
- Which version is current?
- Does the query require exact terms or conceptual similarity?
- How much evidence is enough?
- Which sources should rank higher?
A model cannot correct a missing source it never received.
More context can make the answer worse
Large context windows make it possible to send many documents at once. Possible is not the same as useful.
The paper “Lost in the Middle” tested how models use long inputs. Performance often dropped when the relevant information appeared in the middle of a long context, even for models designed to accept long inputs.
The practical lesson is not that long context is useless. It is that dumping every possible file into a request does not remove the need for retrieval.
Extra material can introduce:
- Old versions.
- Contradictory statements.
- Irrelevant projects with similar names.
- Duplicate passages.
- Content outside the user’s purpose.
The goal is a small, high quality evidence set.
Exact terms and meaning need different search methods
Semantic vector search is good at finding conceptually similar language. It can connect “customer cancellation” with a document that discusses “ending the subscription.”
Keyword search is better for exact items such as product codes, customer names, dates, invoice numbers, and policy identifiers.
Microsoft’s Azure AI Search documentation explains hybrid search as running full text and vector queries together, then merging the results. This gives the system both precision and conceptual reach.
For company knowledge, filters add another layer. Search may be limited to the active customer, approved policy status, a department, or a date range before results are ranked.
Metadata is part of relevance
The text alone does not tell the full story.
A strong retrieval index includes useful metadata:
- Source system.
- Owner.
- Created and updated dates.
- Approval status.
- Customer or project relationship.
- Document type.
- Access groups.
- Version or lifecycle state.
This helps the system prefer the signed contract over a proposal template or the approved policy over an old draft.
Evaluate with real company questions
Retrieval quality cannot be judged from a few convenient demos.
Create a set of questions employees actually ask. For each one, record the evidence a correct answer requires and who should be allowed to see it.
Then test:
- Did the system retrieve the necessary source?
- Did irrelevant material outrank it?
- Was the source current?
- Were permissions preserved?
- Did the answer cite the evidence accurately?
If retrieval fails, changing the writing prompt will not solve the problem.
Use live tools when the state must be current
An index is useful for discovery and fast retrieval. Some answers require a live check.
The system may retrieve the project history from its index, then call the project API to confirm the current assignee before proposing an action. It may find the relevant product from company knowledge, then query inventory live.
This combination provides both context and current state.
Improve the input before replacing the model
When an answer is poor, inspect the full path:
- What query did the system create?
- Which sources were searched?
- What filters were applied?
- Which results were ranked highest?
- What evidence reached the model?
- Which source supported each claim?
This usually reveals a concrete problem the team can fix.
The Business Brain is designed around source backed answers because retrieval quality is part of the product, not invisible preparation.
A larger model can reason more effectively.
Better retrieval gives it something worth reasoning about.
Questions people ask
What is retrieval in a Business Brain?
Retrieval is the process of selecting relevant, permitted evidence from company sources before a model answers a question or plans an action.
Is a large context window enough for company knowledge?
No. Research shows that models do not use every part of long context equally well. Sending more material can add noise, cost, and conflicting evidence.
What is hybrid search?
Hybrid search combines exact keyword search with semantic vector search, then merges or reranks the results. It helps with both precise identifiers and conceptually similar language.