Your knowledge search needs agentic retrieval when people ask questions that one search cannot answer: questions that span two document collections, depend on choosing the right source or need a second look. If most questions come from one set of documents, classic retrieval-augmented generation (RAG) is still the right design. This guide explains the difference, using a recent AWS reference build, and how to roll agentic retrieval out safely.
Where does classic RAG fall short?
Most RAG systems work the same way. They turn the question into an embedding, fetch the closest passages from one index, give them to a model and return the answer. As the AWS team puts it, for a large class of questions that is the right design, and there is no reason to build anything more complicated.
It fails in three recognizable ways. Each one matters to a business because the answer still sounds confident.
- The answer spans two collections. One search returns half of what the model needs, and the answer comes back fluent but incomplete.
- The right source depends on the question. A single index has no way to choose between, say, contracts and support notes.
- The question needs a second look. After seeing the first result, there is nowhere for a follow-up search to happen.
What is agentic retrieval?
Agentic retrieval puts a reasoning agent in the loop. The agent decides whether to search, which knowledge base to consult and whether to go back for more before it writes an answer with citations.
Classic RAG
- Question
- Search one index
- Give passages to the model
- Answer
Agentic retrieval
- Question
- Agent picks a knowledge base
- Search
- Enough context?
- Answer with citations
Source: AWS Machine Learning
In the AWS design, each knowledge base is exposed to the agent as its own tool. Within a knowledge base, the AgenticRetrieveStream API breaks the question into sub-queries, searches repeatedly and returns a grounded answer with citations. The agent then checks whether it has enough context. If not, it routes and searches again.
Should you use a managed or a self-run knowledge base?
Amazon Bedrock now offers a managed knowledge base that handles ingestion, storage, indexing and retrieval, including embedding and reranking. There is no vector database to provision, scale or patch. On Bedrock, agentic retrieval is available only on this managed type.
A customer-managed knowledge base is still the right choice in some cases. AWS names three: when you want to choose and run your own vector store, reuse an existing OpenSearch Serverless collection, or meet specific data-residency, tuning or compliance requirements. If any of those apply, settle them before you choose the agentic path.
How do you roll out agentic retrieval step by step?
Start from the questions, not the technology. Collect the questions staff and customers actually ask, and note which documents answer each one. That list tells you whether you need agentic retrieval at all.
Then build in stages so each step proves itself before the next one adds cost.
- Group documents by topic. Give each domain its own knowledge base so the agent can route. The AWS example keeps financial and weather documents in separate knowledge bases.
- Keep simple questions simple. Answer single-source questions with classic RAG.
- Add routing where answers span sources. Turn on agentic retrieval for the questions your list shows need more than one source.
- Require citations. Every answer should trace back to the passages it used, so people can check it.
- Measure before launch. Set up monitoring and evaluation first, as the next section explains.
Why should monitoring come before launch?
A single-shot pipeline is a function you can understand by reading it. An agent that reasons and searches in a loop can take a different path on every question. By default you cannot see the path it took or whether the answer was any good. AWS’s conclusion is that observability becomes the only way to know the system works.
The AWS reference build provisions two Amazon CloudWatch dashboards covering seven layers of telemetry, from ingestion and retrieval quality to token usage and evaluation scores. It runs evaluation both on demand and continuously. Faithfulness and correctness scores let you catch answers drifting from their sources before users find the problem.
What are the trade-offs?
Agentic retrieval gives better answers on complex questions, but it is a more complex system to operate. Each extra round of planning and searching is more work for the system, so watch latency and token usage per question. You also take on a platform dependency: on Bedrock, the agentic features come with the managed knowledge base.
| Approach | Best for | Watch out for |
|---|---|---|
| Classic RAG | Questions answered by one set of documents | Fluent but incomplete answers when a question spans sources |
| Agentic retrieval | Questions that span collections or need follow-up searches | More moving parts; needs monitoring and evaluation from day one |
| Customer-managed store | Existing vector stores, data residency or tuning needs | On Bedrock, agentic retrieval needs the managed knowledge base |
What should you do next?
Export a month of real questions from your help desk, shared inbox or internal chat, and tag each one with the documents that answer it. If most questions point to one source, improve classic RAG first. If many span sources, plan a small agentic pilot with citations and evaluation scores in place before anyone relies on it. If you are weighing this for your business, see how we approach RAG development and knowledge search and LLM integration, or book a meeting with our team.
Frequently asked questions
Do we need agentic retrieval for an internal FAQ assistant?
Usually not if the answers come from one set of documents. Classic RAG is simpler to run and, as AWS notes, the right design for a large class of questions.
Does a managed knowledge base mean no vector database?
On Amazon Bedrock’s managed type, Bedrock handles storage, indexing, embedding and reranking, so there is no vector database for you to provision. With a customer-managed knowledge base, you choose and run the vector store yourself.
How do we know the answers are accurate?
Require citations so answers trace back to source passages, and track evaluation scores such as faithfulness and correctness. The AWS reference build runs these both on demand and continuously.
Sources
- Build agentic retrieval on a Managed Amazon Bedrock Knowledge Base with CDK, AWS Machine Learning, 2026-10-26
Researched from the sources above with AI assistance and checked by the Aranv team’s publishing standard.