Lesson 2.1: Generative AI fundamentals for solution design
A GenAI feature can look brilliant in a demo and still fail the moment it meets messy documents, live tools, or a user who asks the wrong question. If you design AI solutions, you need to know why models generate the outputs they do, where they break, and which controls actually make them reliable. Those choices affect cost, latency, safety, and whether your system produces useful answers or confident mistakes. With that foundation, you can make smarter decisions about prompts, grounding, structured outputs, and the guardrails your design needs.
How foundation models produce text and structured outputs
A foundation model does not “look up” an answer. It generates the next token that is most probable given the tokens it has already seen, plus any system and user instructions. A token is a chunk of text, often a word piece, and the model’s output is a probability distribution over the next possible token. Generation is the repeated act of sampling from that distribution until a stop condition is reached.
The context window is the model’s working memory for a single request. Everything that matters for the response has to fit inside it: system instructions, user prompt, retrieved documents, tool results, and the model’s own prior output. When the prompt gets long, cost and latency rise because the model must process more tokens. Quality can also degrade because the model has more competing instructions and more opportunities to anchor on irrelevant text.
Decoding strategy controls determinism and variance. Greedy decoding picks the highest probability token each step and tends to be repeatable, but can get stuck in bland or locally optimal phrasing. Sampling with temperature increases diversity by flattening the distribution, which is useful for ideation but risky for compliance and structured outputs. Top-k and nucleus (top-p) sampling constrain sampling to a subset of likely tokens, which can reduce nonsense while still allowing variation.
Structured outputs are still token sequences, so “JSON mode” is not magic. The model can drift into invalid syntax unless you constrain it with explicit schemas, strong formatting instructions, and post-generation validation. In practice, long inputs push you toward summarization and chunking, and strict response formats push you toward schema-first design where the model fills fields rather than inventing structure. Those choices set up the rest of the inference pipeline, including where you enforce safety and correctness.
Common failure modes and why they occur
Your system can look solid in a demo and still fail the first time real users, messy documents, or live tools hit it. These failures usually come from predictable weak points, not random surprises. When you know the common failure modes, you can design controls before they turn into incidents. That gives you a practical way to make your prompts, retrieval flow, and tools safer in real work.
These are the failure modes you will see most often and the main controls you can use to reduce them:
- Hallucinated facts: The model predicts plausible text, so it may invent facts or citations when your prompt is vague or lacks grounding. You can reduce this by grounding answers in retrieved sources, requiring citations tied to chunk IDs, and verifying those citations in code or reviews.
- Prompt injection: This happens when untrusted text sits too close to trusted instructions, so the model cannot reliably tell policy from document content. You can reduce this by separating roles, sanitizing retrieved content, allowlisting sources, and testing with adversarial prompts and documents.
- Format drift: Structured output is still generated token by token, so JSON and schemas can break when your constraints are weak or your prompt asks for too much at once. You can reduce this by defining an explicit schema, using stop sequences, and rejecting outputs that fail parsing or schema validation.
- Unsafe output: Training data contains bias and harmful patterns, so unsafe or skewed content can still appear, especially in edge cases or transformations of unsafe text. You can reduce this by adding safety classifiers, refusal policies, sensitive-topic routing, redaction, and policy-based evaluation.
- Instruction failure: The model may only partly follow your request when you mix narrative guidance with hard constraints or let earlier rules fall out of context. You can reduce this by narrowing the task, reasserting non-negotiable rules in a stable system message, and using explicit output schemas.
- Tool misuse: In agentic systems, the model can pick the wrong tool, send unsafe parameters, or keep calling tools after the task is done. You can reduce this by constraining tool interfaces, validating parameters, adding confirmation steps for risky actions, and checking tool traces instead of only final text.
- Data leakage: Sensitive data can leak when you include secrets, raw logs, or overly broad tool outputs in the context window. You can reduce this by minimizing context, redacting secrets, limiting tool output access, and scanning outputs for keys, canary tokens, or copied sensitive strings.
Architect’s Note: Treat every boundary where untrusted text enters the prompt as an attack surface. Retrieved documents, user-uploaded files, and tool outputs should be handled like inputs to a web app: validate, sanitize, and keep them out of the instruction channel.
| Failure mode | Root cause category | Prevention controls | Detection signals | Validation method |
|---|---|---|---|---|
| Hallucinated facts or citations | Model limitation plus missing grounding | Use retrieval grounding; require citations tied to retrieved chunk IDs; constrain answers to provided sources | Citations that do not match retrieved sources; confident tone with low overlap to context | Automated citation verification; human spot checks on a labeled set |
| Prompt injection via retrieved content | System design limitation | Separate system policy from retrieved text; strip or neutralize instruction-like patterns; allowlist sources | Model repeats “ignore previous instructions”; sudden policy violations after retrieval | Red-team prompts; regression tests with adversarial documents |
| Format drift for JSON or schemas | Model limitation under weak constraints | Provide explicit JSON schema; use stop sequences; post-parse and reject invalid outputs | JSON parse errors; missing required fields; extra unexpected keys | Schema validation in code; contract tests with fixed prompts |
| Biased or unsafe outputs | Training data exposure plus incomplete safety tuning | Safety classifier; refusal policy; sensitive-topic routing; redaction of PII | Toxicity score spikes; protected-class mentions in irrelevant contexts | Safety evaluation set; policy-based unit tests |
| Tool misuse in agentic flows | Control and planning limitation | Tool allowlists; parameter constraints; confirmation steps for destructive actions; max-steps limits | Repeated tool calls; calls with out-of-range parameters; high error rates from tools | Trace-based evaluation; simulated tool environment tests |
| Data leakage from context | System design limitation | Minimize context; redact secrets; do not include raw logs or credentials; least-privilege tool outputs | Output contains tokens resembling keys; verbatim reproduction of sensitive strings | Secret scanning on outputs; canary tokens; privacy test suite |
Workflow patterns for building reliable GenAI features
A reliable GenAI feature starts with a task definition that is testable. “Answer questions about our product” is not testable. “Given a support ticket and the current knowledge base, draft a reply that cites the relevant articles and never invents policy” is testable because you can check citations, policy compliance, and whether the draft is useful. Success criteria should include both quality metrics and safety constraints, because production failures are often policy violations, not low BLEU scores.
Key Takeaway: Treat the model as a probabilistic component inside a deterministic system that constrains inputs, validates outputs, and measures behavior against explicit success criteria.
Your grounding strategy comes next because it sets the limits of what the model can use. If the task depends on current or proprietary information, retrieval is usually the right choice. If the task is stable and repetitive, prompt templates or fine-tuning may be enough, but you still need a plan for fact changes over time. Prompt design should stay simple and deliberate: define the role, constraints, and output format first, keep untrusted text separate from instructions, and validate structured outputs in code.
You also need safety and evaluation throughout the workflow, not just at the end. Check inputs before generation, check outputs after generation, and validate tool use in agentic systems before anything executes. Tie evaluation to the same success criteria you defined at the start, and run those tests whenever you change the prompt, model, or retrieval setup. In production, keep monitoring the same signals so you can catch drift early and improve the system with evidence instead of guesswork.
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
