Lesson 4.4: Analytics services for developers
Analytics in AWS is not just about data science. Many modern applications generate logs, clickstreams, IoT telemetry, or event streams that require processing, transformation, query, and indexing.
For developers, the core skill is understanding which service fits which workload, considering data velocity, query requirements, operational overhead, and scaling behavior. In DVA-style exams or scenario questions, AWS evaluates services primarily based on:
- Ingestion pattern (real-time vs batch)
- Query flexibility (ad hoc SQL vs search)
- Operational overhead (managed vs custom consumers)
Analytics decisions often affect cost, latency, durability, and scalability, so architectural reasoning is key.
Amazon Kinesis: real-time streaming ingestion
Amazon Kinesis is designed for processing streaming data in near real-time. It is particularly effective for continuous event ingestion from applications, IoT devices, or logs.
Key services
- Kinesis Data Streams (KDS)
- Provides shard-based streams for custom real-time processing.
- Each shard preserves order within itself.
- Consumers independently read and process events.
- Shard scaling is manual, requiring monitoring to avoid throttling.
- Kinesis Data Firehose
- Fully managed delivery to S3, Redshift, OpenSearch, or Splunk.
- Buffers records automatically and delivers them without custom consumers.
- Simplifies operational complexity at the cost of flexibility.
Developer considerations
- Data Streams when:
- Low-latency, real-time processing is needed
- Custom business logic is applied per event
- Firehose when:
- Near-real-time delivery to storage or analytics systems is the goal
- Developers want minimal operational overhead
Scaling & monitoring
- Monitor ShardIteratorAge and IncomingRecords metrics to detect lag.
- Adjust shard count for throughput scaling.
- Set up CloudWatch alarms for throttling and processing delays.
Example
Multiple producers sending events to Kinesis shards, and multiple consumers performing processing or analytics in real time. Include Firehose delivering aggregated records to S3 and OpenSearch.
Amazon Athena: serverless SQL queries on S3
Athena is a serverless query engine for analyzing data stored in S3. It uses schema-on-read, meaning no schema is enforced at write time.
Use cases
- Query application logs stored in S3
- Analyze CloudTrail or VPC flow logs
- Run ad hoc analytics without managing a database
Developer workflow
- Register data in the AWS Glue Data Catalog.
- Map structured formats such as CSV, JSON, or Parquet.
- Execute SQL queries directly against S3 objects.
SELECT COUNT(*) AS purchases
FROM clickstream_logs
WHERE page = '/checkout'
AND event_date = '2026-03-02';
Key considerations
- Athena is not designed for transactional workloads.
- Performance improves with columnar formats (Parquet/ORC) and partitioning.
- Queries are serverless and pay-per-query, reducing infrastructure management.
Amazon OpenSearch Service: search and log analytics
OpenSearch Service is optimized for full-text search, log aggregation, and real-time dashboards.
Use cases
- Real-time search within web applications
- Indexing application logs for analytics
- Visualizing operational metrics with dashboards
Developer workflow
- Index documents via API or streaming ingestion (Kinesis/Firehose).
- Perform structured or unstructured queries.
- Visualize with OpenSearch Dashboards (formerly Kibana).
Key considerations
- Optimized for search, not relational transactions.
- Supports horizontal scaling via shards and replication.
- Integrates with CloudWatch for monitoring and alerts.
Example:
A clickstream pipeline:
Kinesis Data Streams -> Lambda -> OpenSearch
- Lambda function transforms click events to JSON documents suitable for indexing.
- OpenSearch indexes the data, supporting queries like “find all checkout events in the last 10 minutes.”
Integrating Kinesis, Lambda, and OpenSearch
A common real-time analytics pattern:
- Kinesis ingests event streams.
- Lambda functions consume Kinesis events:
- Transform data
- Filter or enrich events
- Write to OpenSearch or S3 for storage
- OpenSearch enables dashboards and alerts.
- Athena allows ad hoc querying of S3 raw logs.
import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
os_client = OpenSearch(
hosts=[{'host': 'my-os-domain.us-east-1.es.amazonaws.com', 'port': 443}],
http_auth=('user', 'password'),
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
def lambda_handler(event, context):
for record in event['Records']:
doc = transform(record['kinesis']['data'])
os_client.index(index='clickstream', body=doc)
Architectural Selection Guidance for Analytics
| Requirement | Recommended Service | Notes |
|---|---|---|
| Real-time streaming ingestion | Kinesis Data Streams | Low-latency processing, custom consumers |
| Near-real-time delivery to storage | Kinesis Data Firehose | Fully managed, minimal operational overhead |
| Ad hoc SQL queries on S3 | Athena | Serverless, schema-on-read, pay-per-query |
| Full-text search and log dashboards | OpenSearch | Fast query, real-time indexing |
Key Takeaways for Developers:
- Match data velocity to the service: streaming data goes to Kinesis.
- Match query type: SQL analytics uses Athena.
- Match search & aggregation: dashboards or logs use OpenSearch.
Operational Guidance:
- Monitor metrics and set CloudWatch alarms.
- Partition S3 and optimize query formats for Athena.
- Scale shards or nodes appropriately for Kinesis and OpenSearch.
- Ensure data durability by buffering events in S3 or Kinesis before processing.
