Lesson 2.3: Core architectures including RAG fine tuning and agents
Generative AI concepts and workflows
A generative AI system fails in production for boring reasons. The model answers confidently from stale context, the retrieval layer returns the wrong chunk, or an agent calls a tool with the right intent but the wrong parameters. The fix is rarely “use a bigger model.” It is usually an architectural choice about where truth comes from, how it is injected into the model’s context, and what parts of the system are allowed to take actions.
The three patterns that matter most are retrieval-augmented generation for grounding, fine-tuning for consistent behavior, and agents for multi-step tool use. Each pattern shifts cost, latency, and risk into different parts of the stack, so the right choice depends on what must be correct, what must be fresh, and what must be auditable.
Retrieval augmented generation as a grounding strategy
Retrieval-augmented generation is a system design where the model is not treated as the source of truth. The source of truth is a corpus you control, and the model’s job is to synthesize an answer from retrieved evidence. Architecturally, RAG is a pipeline with distinct failure modes at each stage, which is why it is worth naming the components explicitly.
Document ingestion is where raw content becomes something retrievable. That includes extracting text from PDFs, normalizing encodings, stripping boilerplate, and attaching metadata like document ID, version, authoring system, and access control labels. Chunking then splits documents into retrieval units. Chunk size is not a tuning footnote. It determines whether retrieval returns enough context to answer precisely without flooding the prompt with irrelevant text. Overly small chunks increase recall but often lose the connective tissue needed for correct synthesis. Overly large chunks reduce retrieval precision and can push you into context window limits.
Embeddings convert each chunk into a vector representation, and indexing stores those vectors in a structure optimized for nearest-neighbor search. Retrieval takes a user query, embeds it, and returns the top K chunks by similarity, often filtered by metadata such as tenant, language, or document type. Many production systems add reranking as an optional second stage that uses a more expensive model to reorder candidates based on relevance to the query, which is a direct way to trade latency for answer quality.
RAG reduces hallucinations because the model is constrained by evidence you provide, and it supports freshness because updating the corpus updates the answers without retraining. The trade is operational. Retrieval adds latency, and index consistency becomes a real concern. If ingestion is asynchronous, users can see a document in the UI but not get it in answers for minutes or hours. If you do not track document versions, you can retrieve chunks from an older revision and synthesize an answer that is technically grounded but wrong.
Provenance is the difference between “grounded” and “trustworthy.” If the system can cite the retrieved chunks, users can validate and you can debug. Without citations, you will spend time arguing about whether the model made something up or whether retrieval returned a misleading passage. The next step is making the pipeline visible enough that you can inspect what was retrieved and why, which is where a diagram helps.
Fine tuning and adaptation when prompts and RAG are not enough
Fine-tuning is the move you make when you want the model to behave differently, not when you want it to know more facts. If the problem is missing or changing knowledge, RAG is usually the right lever. Fine-tuning becomes appropriate when you need consistent style, domain-specific language, or structured outputs that must be reliable across many prompts and users.
A common trigger is output format compliance. If you need the model to emit JSON that matches a schema, or to produce a specific classification label set with low variance, prompting can get you part of the way but tends to degrade under edge cases. Another trigger is specialized writing style or tone that must be consistent, such as customer support responses that follow a policy and phrasing guide. Fine-tuning can also help with tool-usage patterns, where the model must learn when to call a tool and how to fill parameters in a predictable way, rather than improvising.
Conceptually, there are two families of adaptation. Full fine-tuning updates a large portion of model parameters, which can produce strong behavior changes but increases cost, training time, and the risk of unintended shifts. Parameter-efficient approaches update a smaller set of weights or add small trainable components, which can be cheaper and easier to iterate on while still steering behavior. The practical point is that adaptation is a software lifecycle commitment. You now own training data, training runs, model versions, and regression testing.
The risks are not abstract. Overfitting shows up as a model that performs well on your curated examples but fails on slightly different phrasing. Catastrophic forgetting shows up as a model that becomes worse at general language tasks because the adaptation pushed it too far toward a narrow distribution. Evaluation complexity increases because you must test both the intended behavior change and the unintended side effects, and you must do it across prompt variants, user segments, and safety constraints.
Governance is where many fine-tuning efforts stall. You need dataset lineage, meaning you can trace every training example back to its source system and approval state. You need consent and usage rights, especially if examples include customer conversations or proprietary documents. You also need a way to remove data later, because “delete this customer’s data” is not compatible with a model that has memorized it. If you cannot meet those requirements, you may be better off with RAG plus stricter prompting and post-processing, even if the model is less elegant.
Exam Insight: Fine-tuning is primarily for changing behavior and output consistency. If the goal is to incorporate new or frequently changing facts, retrieval is usually the correct architecture because it updates answers without retraining.
| Pattern | Best for | Data requirements | Operational complexity | Primary risks | Evaluation focus |
|---|---|---|---|---|---|
| Prompting only | Fast iteration on general tasks, low-stakes assistants, prototyping workflows | No corpus required; prompt templates and a small set of test prompts | Low; version prompts and track model changes | Prompt brittleness, inconsistent formatting, hidden policy violations | Prompt regression tests, format compliance rate, safety checks on representative prompts |
| RAG | Answers grounded in enterprise content, freshness, traceability via citations | Curated corpus, metadata, ACL labels, chunking strategy, embedding generation | Medium; ingestion pipelines, index updates, retrieval tuning, observability | Wrong retrieval, stale index, leakage across tenants, latency spikes | Retrieval precision and recall, citation accuracy, end-to-end answer correctness under corpus updates |
| Fine-tuning or adaptation | Consistent style, domain language, structured outputs, stable tool-call patterns | Labeled examples with lineage and consent; train, validation, and holdout splits | High; training runs, model versioning, rollback, governance, drift monitoring | Overfitting, catastrophic forgetting, memorization of sensitive data | Behavioral metrics on holdout sets, side-effect regression, privacy and memorization tests |
| Agents | Multi-step tasks, tool orchestration, workflows that require state and iteration | Tool schemas, API contracts, permissions model, sandbox environments, traces | High; tool reliability, retries, state management, audit logs, safety controls | Tool error propagation, unauthorized actions, infinite loops, prompt injection via tool outputs | Task success rate, tool-call correctness, authorization enforcement, trace-based debugging and replay |
Agentic systems and tool use boundaries
An agent is a system that turns a model from a text generator into a controller that can plan, call tools, observe results, and iterate. The model is still doing language modeling, but the surrounding orchestration loop gives it the ability to take actions in the world. That loop is where most of the engineering work lives.
At a conceptual level, an agent cycle has five steps. Planning decomposes a user goal into sub-tasks. Tool selection chooses which external capability to use, such as search, database queries, ticket creation, or code execution. Tool execution calls the tool with parameters. Observation reads the tool output and updates the working state. Iteration repeats until a stopping condition is met, such as a final answer, a maximum step count, or a failure that requires escalation.
Multi-step reasoning is constrained by what you can make deterministic. The model can propose a plan, but it can also drift, repeat itself, or pursue a plausible but incorrect path. Tool errors propagate because the model tends to treat tool outputs as authoritative, even when the tool returned partial data or an error message. That is why tool outputs should be structured and explicit about success and failure, ideally with typed fields like status, error_code, and result. If you feed raw logs or HTML back into the model, you are inviting it to hallucinate a “successful” interpretation.
Tool schemas are the contract that keeps the agent honest. A schema should define required parameters, allowed values, and constraints like maximum page size or date ranges. It should also define what the tool returns, including error shapes. When the model is allowed to invent parameters, you get failures that look like reasoning problems but are really interface problems. Authorization must be enforced outside the model. The model can suggest an action, but the orchestrator must check identity, scope, and policy before executing, because a prompt injection that convinces the model to “reset all passwords” is only dangerous if the system lets it.
Agents require stronger testing than single-turn chat because the state space is larger. You need sandboxing for tools that mutate data, so you can run end-to-end tests without touching production systems. You need auditability, meaning you log each step, the tool call, the parameters, the tool response, and the final user-visible output. Without traces you cannot debug why an agent took an action, and you cannot prove later that it did not.
Architect’s Note: Treat every tool output as untrusted input. If a tool returns user-generated content, such as a ticket description or a web page, it can carry prompt injection instructions. The orchestrator should strip or isolate that content, and the model should receive it as data with clear boundaries, not as instructions.
Once you can trust the tool boundary and the trace, you can start evaluating the agent as a system rather than as a model, which is where reliability work becomes measurable instead of anecdotal.
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
