Lesson 3.3: Containers and orchestration with ECS EKS and Fargate
A GenAI system rarely fails because the model weights are wrong. It fails because the glue code around the model drifts across environments, a tokenizer version changes, a native library behaves differently on a new base image, or a “temporary” prompt template becomes a production dependency with no versioning. Containers are the practical unit that keeps these moving parts pinned down while still letting you scale and deploy quickly. They also force you to draw boundaries between components, which is where security, observability, and cost control usually become real.
Containerizing GenAI services and dependency management
A container image is the only packaging format that consistently survives the trip from a laptop to CI to production without turning into a dependency archaeology project.
Dependency management apart from requirements.txt is about controlling the base image, the OS packages, and the build toolchain. In practice, a multi stage Docker build is the cleanest way to separate build time dependencies from runtime.
Configuration belongs outside the image. Treat the image as an immutable artifact and inject environment specific values at runtime through task definitions or Kubernetes manifests. Secrets should never be baked into the image or passed as plain environment variables when you can avoid it. On AWS, the common pattern is AWS Secrets Manager or AWS Systems Manager Parameter Store for secret material, with IAM scoped access from the task role or pod role, and rotation handled independently of deployments.
Versioning needs to cover more than the container tag. A GenAI API adapter might depend on a prompt template, a schema for tool calls, and a retrieval configuration that changes behavior as much as code does. Store those artifacts in a versioned location, for example S3 with object versioning, or a Git tagged repository that is built into the image at a specific commit. When you roll back, you want the whole behavior to roll back, not only the binary.
Most containerized GenAI platforms end up with a small set of service types.
- API adapters terminate HTTP, validate requests, and call model endpoints or internal services.
- Retrieval services own vector search and document filtering, and they usually need careful connection pooling and timeouts because they sit on the critical path.
- Rerankers are often CPU heavy and latency sensitive, so they benefit from separate scaling and isolation.
- Background workers handle embedding generation, document ingestion, and evaluation jobs, and they should be designed to tolerate retries and partial failures because they are where backlogs and cost spikes show up first.
| Pattern | Trigger | Typical GenAI Use Case | Failure Handling Approach | Observability Signals to Capture |
|---|---|---|---|---|
| Synchronous API orchestration | Amazon API Gateway to Lambda | Prompt assembly, lightweight retrieval lookup, Bedrock invocation, output shaping | Map model and validation errors to stable HTTP codes, return correlation ID, avoid automatic retries at the edge | p50 and p95 latency, cold start count, Bedrock invocation latency, token usage per request if available, 4xx vs 5xx rate |
| Buffered async ingestion | S3 ObjectCreated to SQS to Lambda | Document ingestion where parsing and chunking are delegated to downstream jobs | SQS redrive to DLQ, idempotency key on object version, exponential backoff for throttling | Queue depth, age of oldest message, Lambda concurrency, DLQ message count, per object processing time |
| Fan out evaluation | Scheduled EventBridge rule to Lambda | Nightly prompt regression tests, model comparison runs, safety checks on a fixed dataset | Write each test case result independently, retry transient model errors with jitter, store partial progress | Success rate by test suite, cost per run, token usage distribution, per model latency, error codes |
| Orchestrated multi step workflow | Step Functions state machine with Lambda tasks | RAG pipeline with retrieval, reranking, model call, post processing, and human review branch | Step Functions retries with per state policies, catch blocks to route to remediation, timeouts per step | State transition counts, per state duration, retry counts, failure causes, end to end duration |
| Streaming response proxy | API Gateway or Lambda Function URL to Lambda with streaming | Chat style UX where tokens stream to the client while applying lightweight filtering | Abort on policy violation, timeouts tuned for streaming, circuit breaker on downstream throttling | Time to first token, stream duration, client disconnect rate, throttling events, partial completion rate |
Choosing among Amazon ECS Amazon EKS and AWS Fargate
The decision is usually not “containers or not.” It is whether you want Kubernetes as an API surface, and whether you want to own the worker nodes. Amazon ECS is a good default when the workload is a set of services and workers that can be expressed as tasks, and the team wants fewer moving parts. Amazon EKS earns its cost when you need Kubernetes ecosystem integrations, custom controllers, or you already have standardized Kubernetes operations and policy enforcement.
Operational overhead is the first real constraint. EKS gives you more flexibility, but it also gives you more places to create outages. ECS has fewer of those failure modes because the control plane is narrower. If the platform team is small, or if GenAI is one product among many, ECS often produces a more predictable operational posture.
Compute choice is the second constraint. AWS Fargate removes node management, which is valuable for stateless API adapters and bursty workers. It is a poor fit for GPU workloads because GPU support is not generally available on Fargate, and even when serverless options exist elsewhere, you still need to think about driver compatibility and device plugins. For GPU backed inference services, ECS on EC2 or EKS on EC2 is the common path because you can pick instance families, control AMIs, and validate CUDA and driver versions as part of your release process.
Integration patterns tend to converge. For public entry points, Amazon API Gateway is often used for request validation, auth, and throttling, then forwarded to an internal Application Load Balancer that targets ECS services or EKS ingress. For service to service calls, keep it boring: HTTP with timeouts, retries with jitter, and explicit circuit breaking at the client. If you need discovery and consistent traffic management on ECS, Service Connect can reduce the amount of bespoke glue. On EKS, a service mesh can do the same, but it is another operational system that must justify itself.
Architect’s Note: Autoscaling GenAI services based only on CPU is a common way to get surprised. For API adapters, scale on request rate and p95 latency. For workers, scale on the queue depth and the age of the oldest message. For retrieval services, watch connection pool saturation and downstream timeouts. If the scaling signal does not map to user-visible pain or backlog growth, it will not behave the way you want under load.
The cleanest approach for many teams is a mixed model: keep the latency-sensitive API tier on ECS or EKS with conservative scaling and strict rollout controls, and run embedding and evaluation workers as separate services with aggressive scaling and hard cost limits. That separation makes the next step, standardizing observability and deployment safety, much easier.
Network and encryption controls including VPC endpoints and KMS
In containerized GenAI systems, identity alone is not a sufficient control. A correctly scoped IAM role can still leak sensitive data if the workload has unrestricted network access. The system must enforce where data is allowed to flow, not just who can access it. The goal is to make intended communication paths explicit and constrained, while treating all other paths as exceptions.
Application workloads should run in private subnets by default, with outbound internet access treated as a controlled exception rather than a baseline capability. For AWS-native dependencies, private connectivity through VPC endpoints ensures that traffic remains on the AWS network and can be restricted using endpoint policies. Services such as Amazon S3 and AWS KMS are central to most GenAI pipelines, particularly for retrieval and storage, and should be accessed through these endpoints. When using S3 gateway endpoints, combine them with bucket policies that enforce aws:sourceVpce, so access is only permitted through the intended network path.
Model invocation introduces a similar consideration. Where private endpoints are supported, they should be used to constrain access to specific APIs and resources. Where they are not available, outbound traffic should be routed through a controlled layer such as a NAT gateway or egress proxy. Security groups should allow only required destinations, and broader outbound access should be mediated through systems that enforce allow lists and produce audit logs. This reduces incident response complexity by limiting analysis to a well-defined set of network paths rather than the entire application surface.
Encryption must be applied consistently across all layers of the system. At rest, prompts, embeddings, corpora, and logs should be encrypted because any of them may contain sensitive user data. AWS KMS customer-managed keys provide control over access boundaries, cross-account usage, and key lifecycle. The key policy becomes the primary control surface and should grant only the minimum required permissions. Application roles interacting with encrypted S3 data typically require kms:Decrypt and kms:GenerateDataKey, and those permissions must be explicitly allowed in both IAM and the key policy.
Encryption in transit should be enforced rather than assumed. For S3, bucket policies should deny non-TLS access using aws:SecureTransport. Internal service communication should use TLS endpoints with certificate validation wherever supported. A common oversight in GenAI systems is treating observability data as less sensitive than primary data. Logs and traces frequently contain prompt content, making them subject to the same security requirements. CloudWatch Logs and other observability stores should be encrypted with KMS and configured with retention policies aligned to compliance needs.
Once network boundaries and encryption controls are enforced, governance becomes a matter of visibility and traceability. The system should make it possible to determine what data exists, where it flows, and which components accessed it. This is what turns infrastructure controls into enforceable guarantees rather than assumptions.
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
