Lesson 3.1: Selecting EC2 for GPU and ML inference workloads
A GenAI system usually fails on the boring parts first. The model might be accurate, but the service still times out because token generation is slower than expected under concurrency, or because the embedding job saturates storage throughput and starves the GPU. Amazon EC2 is the place you end up when you need explicit control over accelerators, memory bandwidth, and network behavior, or when you need to run a custom inference stack that does not fit a managed endpoint. The hard part is not picking “a GPU instance.” The hard part is translating model shape and traffic shape into a compute, storage, and network design that stays predictable under load
How to map workload characteristics to EC2 choices
Instance selection starts with the workload shape. A batch embedding job cares about throughput per dollar and can tolerate queueing. A real time chat endpoint cares about tail latency under concurrency, which is usually dominated by token generation speed, memory bandwidth, and how well the runtime batches requests. Those two workloads can use the same model and still want different instance families, different scaling policies, and different placement decisions.
Batch versus real time is the first cut because it determines whether you optimize for utilization or for response time.
- Batch jobs can run at high GPU utilization and accept occasional retries.
- Real time inference needs headroom because a single slow request can pin a worker and amplify queueing.
- If you are serving streaming responses, you also need to think in terms of tokens per second per concurrent session, not just requests per second.
Concurrency and model size drive memory decisions. Large language models and large embedding models are often constrained by accelerator memory and memory bandwidth before they are constrained by raw compute. If the model does not fit in accelerator memory, you are forced into sharding, offloading, or smaller variants, and each of those choices changes latency and operational complexity. Even when the model fits in GPU memory, performance can still be limited by memory bandwidth. In this case, the GPU appears underutilized because it is waiting for data to move between memory and compute units. This results in higher latency even though compute capacity is available.
Network throughput matters earlier than most teams expect. If you are doing retrieval augmented generation, the inference node is not only generating tokens. It is also calling a retriever, fetching context, and sometimes calling other internal services. If those calls cross Availability Zones or traverse a NAT path, you pay in latency and in variance. For multi-node training or distributed inference, you also need to consider whether the instance family supports higher bandwidth networking and whether placement groups are required to keep east west latency low.
GPU instances are the default for custom inference, embedding generation, and any workload where the model runtime is the bottleneck. CPU is often sufficient for orchestration, document parsing, chunking, and lightweight feature extraction, especially when those steps are I/O bound. A common pattern is to keep preprocessing on CPU instances and reserve GPU capacity for the steps that actually benefit from acceleration. That separation also makes scaling cleaner because the CPU tier and GPU tier scale on different signals.
Scaling and capacity planning should be explicit about what you are scaling. For batch, scale on queue depth and age of oldest message, then cap concurrency per worker to avoid thrashing the accelerator. For real time, scale on p95 latency, in flight requests, and accelerator utilization, but treat utilization as a lagging indicator because it can look healthy while tail latency degrades. Placement groups can reduce jitter for tightly coupled workloads, but they also reduce placement flexibility, so they can increase capacity errors during scale out. That trade is acceptable when latency is the product, and painful when throughput is the product.
| Instance Family | Primary Accelerator | Best Fit GenAI Tasks | Key Bottleneck to Watch | Cost Control Lever |
|---|---|---|---|---|
| P5 | NVIDIA H100 GPU | High throughput LLM inference, large batch embedding generation, distributed training style workloads that benefit from very high compute and memory bandwidth | Accelerator availability and scaling limits, plus diminishing returns if the pipeline is I O bound | Use Savings Plans for steady state, and keep the GPU fed by staging data in S3 and using local NVMe where appropriate |
| P4d or P4de | NVIDIA A100 GPU | LLM inference and embedding generation when H100 is not required, multi GPU workloads that benefit from high network bandwidth | Memory capacity per GPU and interconnect behavior under multi GPU sharding | Right size by measuring tokens per second per dollar, and use Auto Scaling with warm pools if startup time is significant |
| G5 | NVIDIA A10G GPU | Cost sensitive inference, smaller LLMs, image generation variants that fit well on mid range GPUs, development and staging | GPU memory and memory bandwidth, plus CPU to GPU balance for preprocessing heavy stacks | Use Spot for batch embedding jobs, and cap per instance concurrency to avoid latency spikes |
| G6 or G6e | NVIDIA L4 or L40S GPU | Efficient inference for smaller models, embedding services, and mixed workloads where power efficiency matters | PCIe and memory bandwidth limits for larger models, plus runtime batching efficiency | Prefer these for always on endpoints when the model fits comfortably, then scale horizontally with smaller nodes |
| Inf2 | AWS Inferentia2 | High volume inference where the model can be compiled to Neuron, especially for stable architectures and predictable throughput | Compilation and operator support constraints, plus model portability across runtimes | Commit to Neuron only when the model and framework are stable, then drive utilization with batching and steady traffic |
Data and network design for embedding and retrieval pipelines
Embedding generation is usually a data movement problem disguised as a compute problem. The GPU can generate vectors quickly, but only if documents arrive at a steady rate and the worker can write results without blocking. Amazon S3 is a good staging layer because it decouples producers from consumers and gives you durable, cheap storage for raw documents and intermediate artifacts. The design question is how you move data from S3 to the worker fleet without turning S3 reads into the bottleneck.
For ingestion jobs, plan for parallel reads and predictable object sizing. Many small objects increase request overhead and can make throughput spiky. A common approach is to store documents in moderately sized objects, then have workers read ranges or batches, parse, chunk, and embed. If you need very high throughput, consider caching hot inputs on instance store or EBS, but treat that cache as disposable. The source of truth stays in S3 so that retries and reprocessing are simple.
Egress and cross boundary traffic show up in both cost and latency. If your embedding workers run in a VPC and pull from S3 over a gateway endpoint, you avoid NAT data processing charges and keep traffic on the AWS network. If you instead route through a NAT gateway, you pay per GB and you add another moving part that can throttle under load. The same logic applies to calling downstream services like a vector database or an internal metadata service. Private connectivity is not only a security preference. It is a performance control.
Retrieval augmented generation is sensitive to latency variance because retrieval sits on the critical path. If the retriever call sometimes takes 20 ms and sometimes takes 200 ms, the user experiences that as “the model is slow,” even if token generation is fast. Keep the inference tier and the retrieval tier in the same Region and usually the same Availability Zone when possible. If you need multi AZ for availability, make the retrieval client aware of zone local endpoints so that it prefers local reads and only fails over across zones when necessary.
Bandwidth affects retrieval quality in indirect ways. If the system is bandwidth constrained, teams often reduce the number of retrieved chunks or shrink context windows to keep latency down. That can degrade answer quality and increase hallucination risk because the model sees less grounding context. It is better to provision enough network and storage throughput so you can retrieve the right amount of context, then optimize the retriever and prompt strategy on top of a stable infrastructure baseline.
Operational controls for EC2 based GenAI components
Operational discipline is what makes EC2 viable for GenAI services at scale. Without tight feedback loops, GPU fleets drift into low utilization, noisy latency, and unpredictable cost. The first control is choosing scaling signals that reflect user impact. For batch embedding, queue depth and age of oldest message are usually better than GPU utilization because utilization can be high even when the system is falling behind. For real time inference, p95 and p99 latency, in flight requests, and error rates should drive scaling, with utilization used to validate that the fleet is not over provisioned.
Health checks need to reflect the model runtime:
- A worker can be “up” but stuck in a bad state, such as a wedged CUDA context, a full disk, or a model server that is accepting connections but timing out internally.
- Use an application level health endpoint that verifies the model is loaded and can complete a small inference within a bounded time.
- For Auto Scaling groups behind a load balancer, wire that health endpoint into target group health checks so instances are replaced when they stop being useful.
Deployments for inference services should minimize cold start and avoid mixing model versions in a way that breaks caching and batching. Blue green works well when you can run two fleets and shift traffic gradually, either by swapping target groups or by weighted routing at the load balancer. If model load time is long, consider warm pools or pre baked AMIs with the runtime dependencies installed, then pull model artifacts from S3 at boot with integrity checks. Rolling updates can work, but only if you control concurrency and drain connections so you do not drop in flight streaming responses.
Logging and metrics should be designed around debugging latency and cost. Push structured logs that include request id, model version, prompt and completion token counts, and timing breakdowns for retrieval, preprocessing, and generation.
In Amazon CloudWatch, track GPU utilization, GPU memory usage, CPU utilization, network bytes, and disk throughput, then correlate those with p95 latency and error rates. Right sizing becomes a measurement exercise: if GPU memory is consistently near the limit, you need a larger accelerator or a smaller model. If GPU utilization is low while CPU is pegged, the bottleneck is preprocessing or the runtime is not batching effectively.
Security boundaries are part of operations because misconfigurations show up as outages:
- Use least privilege instance profiles with narrowly scoped IAM policies, and prefer task specific roles if you are using multiple services from the same instance.
- Encrypt data at rest in S3 and on EBS, and use TLS for service to service calls, especially to retrieval stores and metadata services.
- When private connectivity is available, use VPC endpoints so traffic does not traverse the public internet and so you can apply endpoint policies to constrain what the instance can reach.
Architect’s Note: NAT gateways are a common hidden cost in EC2 based GenAI pipelines. If embedding workers pull from S3 or call AWS APIs through a NAT, you pay per GB and you introduce a shared throughput choke point. An S3 gateway endpoint and interface endpoints for dependent services usually reduce both cost and latency variance.
The operational goal is a fleet that scales on the signals that matter, deploys without long periods of degraded performance during updates, and makes cost visible as a function of tokens and vectors produced. Once those controls are in place, the remaining work is tuning the model runtime and retrieval strategy against a stable compute baseline.
Key Takeaway: Pick EC2 accelerators by matching model and traffic shape to memory, bandwidth, and scaling behavior, then design data paths and operational controls so the GPU spends its time generating value instead of waiting on I/O or recovering from drift.
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
