Lesson 3.3: Event-driven messaging and orchestration for AWS developers
Event-driven architecture is central to modern AWS applications. Services like Amazon SQS, Amazon SNS, Amazon EventBridge, and AWS Step Functions each solve distinct problems in message delivery, processing guarantees, and orchestration. Developers must reason about retry behavior, ordering constraints, scaling implications, and failure modes, rather than just picking a service.
The following architectural diagram illustrates distinct modern application integration patterns, combining these services into a centralized workflow based on EventBridge rules:
Amazon SQS: durable queues and controlled processing
SQS decouples producers and consumers by storing messages durably and enabling asynchronous processing.
- Standard queues: provide at-least-once delivery, best-effort ordering, high throughput.
- FIFO queues: provide exactly-once processing semantics and strict ordering. Deduplication ensures that messages are not processed more than once within the deduplication window, though duplicates may still occur in rare edge cases. FIFO queues have slightly lower throughput compared to standard queues due to these guarantees.
CLI example:
This command creates a FIFO queue named “Orders.fifo” to guarantee message ordering and deduplication for processing critical workflows.
# Create a FIFO queue
aws sqs create-queue --queue-name Orders.fifo --attributes FifoQueue=true
This command attaches a dead-letter queue to the FIFO queue to isolate messages that repeatedly fail processing, preventing infinite retries.
# Configure a DLQ for repeated failures
aws sqs set-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT_ID/Orders.fifo \
--attributes RedrivePolicy='{"deadLetterTargetArn":"arn:aws:sqs:us-east-1:ACCOUNT_ID:OrdersDLQ","maxReceiveCount":"5"}'
Amazon SNS: fan-out distribution
SNS is a publish/subscribe service that broadcasts messages to multiple subscribers. While it does temporarily store messages and performs delivery retries, its primary role is real-time fan-out rather than long-term storage or delayed processing like SQS.
For example, if an application publishes an order event, multiple services such as a payment processor, notification system, and analytics pipeline can all react independently. To create this infrastructure in AWS, we can use:
aws sns create-topic --name OrderEvents
This command creates an SNS topic named OrderEvents to which multiple subscribers can attach. To link an SQS queue to this topic (so that messages are durably processed), you would run:
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:ACCOUNT_ID:OrderEvents \
--protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:ACCOUNT_ID:Orders
This configuration delivers messages to the Orders queue, allowing downstream consumers such as Lambda to process them asynchronously and durably.
Amazon EventBridge: event routing and filtering
EventBridge provides an event bus for rule-based routing of events across AWS services. It supports schema discovery, event replay, and cross-account event routing, which are common in large-scale architectures. Unlike SNS, which only supports subscription filter policies based on message attributes, EventBridge allows filtering based on the entire event payload, enabling more fine-grained, centralized, and decoupled event-driven architectures.
For example, you may want only “order.created” events to trigger a processing Lambda. You can define a rule like this:
aws events put-rule \
--name ProcessOrdersRule \
--event-pattern '{
"source": ["app.orders"],
"detail-type": ["OrderCreated"]
}'
This creates a rule matching events from the app.orders source. Next, attach a Lambda function as the target:
aws events put-targets --rule ProcessOrdersRule \
--targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:ProcessOrders"
Now, only matching events trigger the ProcessOrders Lambda, leaving unrelated events untouched.
AWS Step Functions: workflow orchestration
While SQS, SNS, and EventBridge manage communication, Step Functions coordinate multi-step workflows with retries, parallel execution, and conditional branching. Step Functions are ideal when a process involves multiple dependent tasks, error handling, or long-running operations that exceed Lambda’s 15-minute timeout or require long-running orchestration.
For example, consider processing an e-commerce order. The workflow may require the following steps:
- Payment verification: Confirm the customer’s payment is valid.
- Inventory update: Deduct purchased items from stock.
- Shipment confirmation: Trigger shipping or notify fulfillment systems.
Each of these steps can be implemented as a Lambda function, and Step Functions handles their sequencing, parallel execution if needed, retries, and error branching. You can create the workflow using the AWS CLI:
aws stepfunctions create-state-machine \
--name OrderProcessingStateMachine \
--definition file://order_workflow.json \
--role-arn arn:aws:iam::ACCOUNT_ID:role/StepFunctionsExecutionRole
This command creates a Step Functions state machine that orchestrates multiple Lambda tasks. The workflow logic is defined in order_workflow.json using Amazon States Language (ASL), the JSON-based language for AWS Step Functions. ASL specifies which steps run sequentially, which run in parallel, and how errors are handled. For example, if payment verification fails, Step Functions can retry the Lambda or route the workflow to a failure state, preventing downstream tasks from executing incorrectly.
A typical workflow illustration would show:
- Sequential flow: Payment → Inventory → Shipment
- Retries and error handling: loops or branches for failed payment or inventory errors
- Optional parallel tasks: sending confirmation emails or updating analytics while shipping
This approach ensures reliable, observable, and maintainable workflows, letting developers focus on business logic rather than manually coordinating asynchronous tasks.
Developer responsibilities
Designing resilient event-driven systems requires awareness of runtime behavior:
- Duplicate messages: SQS delivers at least once, so processing must be idempotent.
- Timeouts: When using SQS with Lambda, set the queue’s Visibility Timeout to at least six times the function’s timeout. This prevents messages from being retried before processing completes.
- Retry bursts: Downstream services must tolerate sudden spikes.
- Ordering: FIFO SQS queues enforce strict sequence only when necessary.
- Workflow boundaries: Orchestration logic belongs in Step Functions, not chained Lambdas.
A common failure pattern illustrates the consequences of neglecting these principles:
Lambda fails → SQS retries → downstream database overwhelmed → cascading failure
Correct architecture requires idempotency, proper timeout and retry alignment, DLQs, and concurrency control.
Architectural decision framework
| Use Case | Recommended AWS Service |
|---|---|
| Decoupled async processing | SQS |
| Strict ordering | FIFO SQS |
| Fan-out to multiple subscribers | SNS |
| Rule-based event routing | EventBridge |
| Multi-step orchestration | Step Functions |
My name is Naeem ul Haq. I’ve been working with AWS since its early days and have deep expertise across its evolving ecosystem.
