Lesson 4.2: RAG with Knowledge Bases and vector store options
A RAG system usually fails long before the model sees a prompt. The failure starts when the corpus is messy, duplicated, or chunked in a way that destroys meaning, and the vector store faithfully returns the wrong context with high confidence. Amazon Bedrock Knowledge Bases is essentially an ingestion and retrieval pipeline that turns your documents into embeddings plus metadata, then uses that index at runtime to fetch grounding context for generation.
Knowledge Bases ingestion and document preparation
A common setup is an Amazon S3 bucket as the source of truth. The ingestion job reads objects, normalizes content into a consistent text representation, chunks it, generates embeddings, and writes vectors plus metadata into the configured vector store. The practical implication is that ingestion is not a one-time step. It is a production data pipeline with the same expectations as any other: deterministic inputs, repeatable transforms, and observable outputs.
Document normalization is where retrieval quality is won. PDFs with headers and footers repeated on every page, HTML with navigation boilerplate, and exported wiki pages with duplicated sidebars all create high-frequency tokens that dominate embeddings.
Strip boilerplate, collapse whitespace, and standardize encodings before ingestion. If you cannot normalize perfectly, at least extract stable metadata such as source_uri, doc_id, section_title, last_modified, and access_tier so you can filter and debug later.
Chunking strategy should match how the content is consumed:
- For procedural docs, chunks aligned to headings and steps tend to retrieve better than fixed-size windows because the chunk boundary preserves intent.
- For dense reference material, fixed-size chunks with overlap can work well, but overlap should be justified. Too little overlap drops key definitions that sit on boundaries, while too much overlap inflates index size and increases near-duplicate hits that crowd out diverse results.
- A good default is to chunk by semantic boundaries first, then enforce a maximum size with a small overlap only when needed.
Metadata extraction and filtering tags are the control plane for relevance. If your corpus spans products, regions, or customer tenants, encode that as metadata and require it at query time. Without it, the retriever will happily return the most semantically similar chunk across the entire corpus, which is often the wrong tenant or the wrong version. Treat metadata as part of the contract: define allowed keys, types, and cardinality, then validate them during ingestion.
Data validation should be automated and boring. Run schema checks on metadata, reject empty or near-empty documents, and deduplicate aggressively using stable identifiers and content hashes. Low-quality content is not neutral. It actively competes for retrieval slots and increases hallucination risk because the model is forced to reconcile conflicting contexts. Once the corpus is clean and consistently chunked, the vector store becomes a predictable component instead of a mystery box.
Architect’s Note: If you do not version your documents and propagate version or effective_date into metadata, you will eventually retrieve outdated policy text that is semantically similar to the current version. The model will not know which one is authoritative unless you filter or rank by recency.
Clean ingestion sets the ceiling for retrieval quality, but the vector store backend determines how you scale, isolate tenants, and combine lexical and vector signals when similarity alone is not enough.
Vector store choices and design tradeoffs
The vector store is not just a place to put embeddings. It defines your query capabilities, your isolation model, and how painful operations will be when the corpus grows or tenants multiply. Bedrock Knowledge Bases can integrate with multiple backends, and the right choice depends less on raw similarity search and more on the surrounding requirements: hybrid search, metadata filtering, lifecycle management, and operational ownership.
- Amazon OpenSearch Service is the most natural fit when you need search features beyond pure vector similarity. It supports rich filtering, scoring, and hybrid patterns that combine lexical relevance with vector similarity. It also fits teams that already run search clusters and understand shard sizing, index templates, and query tuning. The tradeoff is operational complexity. You own capacity planning, index lifecycle policies, and the failure modes of a distributed search system.
- Amazon Aurora PostgreSQL with pgvector is a strong option when your application already depends on relational data, and you want transactional control over metadata and access rules. It is often easier to enforce multi-tenant constraints with row-level security or explicit tenant keys, and joins can be useful when retrieval needs to respect business state stored in tables. The tradeoff is that you are building a search system on a database. You need to think about index types, vacuum behavior, connection management, and how vector search performance changes as the table grows.
- Amazon DynamoDB is attractive when you want a serverless operational model and predictable scaling characteristics, especially for key-based access patterns. It can work well when your retrieval layer is tightly constrained, for example, when you always retrieve within a tenant and a document family, and you can design partition keys that keep queries narrow. The tradeoff is that DynamoDB is not a general-purpose search engine. If you need complex ranking, flexible hybrid search, or ad hoc filtering across many attributes, you will feel the constraints quickly.
Selection criteria should start with query patterns:
- If you need a hybrid search because users paste exact error messages, configuration keys, or log fragments, OpenSearch is usually the cleanest path.
- If you need strict transactional governance over what can be retrieved, Aurora can be compelling because access rules can live close to the data.
- If you need minimal ops and your retrieval can be expressed as narrow, well-partitioned queries, DynamoDB can be viable.
Partitioning and index lifecycle are not optional details. Plan for re-embedding when you change embedding models, chunking rules, or normalization logic. That implies either parallel indexes with a cutover strategy or a rebuild window where retrieval quality may degrade. Also plan for deletion. If documents can be revoked, you need a reliable way to remove or tombstone all chunks derived from a source doc_id, and you need to verify the index no longer returns them.
The backend choice sets the constraints, but retrieval quality still depends on how you use it. Once the store is selected, the next step is tuning embeddings and retrieval so that the right chunks win consistently.
| Vector Store | Strengths | Limitations | Best Fit Use Cases | Key Considerations |
|---|---|---|---|---|
| Amazon OpenSearch Service | Strong search and hybrid retrieval, flexible scoring, rich filtering | Operational overhead, cost tied to provisioned capacity | Large corpora, keyword and semantic search | Index tuning, shard sizing, latency monitoring |
| Amazon Aurora PostgreSQL (pgvector) | Relational and vector in one system, strong governance | Not optimized for large-scale vector search | RAG with joins, structured data and access control | Index choice, table growth, read scaling |
| Amazon DynamoDB | Serverless, predictable scaling, low ops overhead | Limited search and ranking flexibility | Tenant-scoped, key-based retrieval patterns | Partition design, hot keys, capacity planning |
| Mixed approach | Best tool per function, flexible and scalable | Higher system complexity, sync required | Large-scale or regulated systems | Data sync, rebuild strategy, consistency checks |
Retrieval tuning including hybrid search and embeddings
The model can only answer from what you retrieve, and retrieval is a set of knobs that trade recall, precision, latency, and cost. The goal is to reliably fetch the few similar chunks that contain the answer, plus enough surrounding context that the model can use them without inventing glue.
Embedding model choice is the first knob. Different embedding models capture different notions of similarity, and the best choice depends on your corpus and queries.
- If your users ask short, keyword-heavy questions, embeddings alone can miss exact matches that matter.
- If your corpus is highly technical, embeddings that handle code, identifiers, and structured text tend to behave better.
Dimensionality matters conceptually because higher-dimensional vectors can represent nuance but can also increase storage and compute costs, so treat it as a capacity planning input rather than a quality guarantee.
Top-k selection controls how much candidate context the model receives, making it a key lever for balancing recall and precision. A well-chosen k ensures the retriever returns enough relevant chunks to cover the answer without overwhelming the model with noise.
If k is too small, relevant context may be missed when queries are ambiguous or chunking is imperfect. If k is too large, loosely related content increases latency and can degrade answer quality because the model must reconcile conflicting signals.
A practical approach is to start with a moderate k that consistently captures the correct context in evaluation, then introduce reranking when higher precision is required. Reranking can refine results by selecting the most relevant passages, but it should be measured because it adds cost and latency.
Hybrid search is worth using when lexical signals carry meaning that embeddings smooth over. Error codes, API names, configuration keys, and quoted log lines are often better handled with lexical matching, while natural language questions benefit from vector similarity. Hybrid search combines both so that exact matches can surface even when the surrounding semantics are broad. In OpenSearch, this typically means composing a query that blends a text query with a kNN vector query and tuning the weights so neither dominates across all query types.
Metadata filters are the second most common source of quality issues, usually because they are missing or too permissive. If the retriever is allowed to search across all products, all regions, and all document versions, it will return something that looks similar even when it is operationally wrong. Filters should be treated as required inputs, not optional hints. If the application cannot determine the right filters, it should ask a clarifying question or fall back to a safe response rather than retrieving across the entire corpus.
Troubleshooting should start with observable artifacts:
- When recall is low, inspect the chunks around the expected answer and verify that chunk boundaries did not separate the key sentence from its definition or prerequisite.
- When hits are irrelevant, check whether boilerplate text is dominating embeddings or whether metadata filters are missing.
- When results look stale, verify ingestion freshness by comparing S3 object LastModified to the index update time, and confirm that ingestion jobs are running and succeeding. Stale indexes are especially tricky because the system still returns plausible context, just not the current truth.
Exam Insight: Hybrid search is not a luxury feature. For corpora with identifiers and exact strings, pure vector similarity can systematically miss the right chunk even when it exists, because the embedding space does not preserve exact token equality.
Once retrieval is tuned, the remaining work is operational discipline: track retrieval hit rates, measure answer grounding, and treat ingestion and indexing as first-class production pipelines rather than background jobs.
Identity and access control for model and data interactions
Least privilege on AWS begins with AWS Identity and Access Management (IAM) roles that separate build time from run time. Developers should not use the same role that production compute uses to invoke models or read embeddings. A practical split is: a CI role that can deploy infrastructure, a developer role that can read non production datasets and logs, and a runtime role attached to the service that handles user traffic. That runtime role is the one that needs tight permissions because it is the one exposed to untrusted inputs.
For model invocation, scope permissions to the exact service and model resources. With Amazon Bedrock, prefer allowing only the specific bedrock:InvokeModel or bedrock:InvokeModelWithResponseStream actions needed by the application, and restrict the Resource to the model or inference profile ARNs you actually use. Model selection should be server side configuration, not a user controlled parameter.
Data access needs the same discipline. For Amazon S3, grant s3:GetObject only on the specific bucket prefixes that contain retrieval corpora, and avoid s3:ListBucket unless the code truly needs it. For vector stores, the principle is identical even if the control surface differs. If you use OpenSearch, scope index access. If you use a managed vector database, scope the API key or IAM integration to the specific collection. The runtime role should not have write access to the corpus unless the product explicitly supports user contributed content and you have a moderation and provenance plan.
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
