Summary:
- AWS Kinesis is a fully managed suite for real-time data streaming. In recent years, AWS rebranded members of this family: Kinesis Data Firehose is now Amazon Data Firehose, and Kinesis Data Analytics is now Amazon Managed Service for Apache Flink.
- The maximum record size for Kinesis Data Streams remains strictly 1 MB. Recent updates focus on KCL 3.x (released late 2024) which introduces automated capacity management and up to 33% compute cost savings, IPv6 support, and on-demand capacity mode enhancements. Long-term data retention up to 365 days has been supported since 2020.
- Understanding shard architecture, enhanced fan-out, partition key strategies, and ordering guarantees during resharding operations is essential for building resilient, high-throughput streaming applications.
- Migration from deprecated KCL 1.x and KPL 0.x requires careful planning, as end-of-support timelines demand immediate action for production workloads.
Real-time data processing has shifted from a competitive advantage to an operational necessity. Whether you are ingesting millions of clickstream events, processing IoT sensor telemetry, or building fraud detection pipelines that must respond in milliseconds, the architecture decisions you make around streaming infrastructure will define your system’s scalability ceiling and operational cost profile. Amazon Kinesis stands at the center of AWS’s real-time data strategy. The 2025-2026 feature releases have fundamentally expanded what architects can achieve with managed streaming services. This guide provides a comprehensive technical examination of the Kinesis ecosystem, covering architectural internals, capacity planning trade-offs, migration pathways, and the practical knowledge required for both production deployments and technical interviews.
The following diagram illustrates the high-level architecture of the AWS Kinesis ecosystem and how data flows from producers through the various Kinesis services to downstream consumers and storage destinations.
Understanding the AWS Kinesis service family
Amazon Kinesis historically encompassed four services, but AWS has unbundled and rebranded parts of the portfolio to align with open-source and standalone engineering standards. The real-time streaming suite now comprises: Kinesis Data Streams, Amazon Data Firehose (formerly Kinesis Data Firehose), Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics), and Kinesis Video Streams.
The architectural distinction between these services centers on the latency-complexity trade-off. Standard consumers using the standard polling mechanism (GetRecords) experience propagation latencies between 200 milliseconds and 1 second. Enhanced Fan-Out drops this latency down to ~70 milliseconds by pushing data over HTTP/2. Firehose abstracts consumer management entirely but introduces buffering latency of 60 seconds minimum (or 1 MB of data).
| Service | Typical latency | Primary use case | Scaling model | Consumer management |
|---|---|---|---|---|
| Kinesis Data Streams | 70ms (Enhanced Fan-Out) / 200ms–1s (Standard) | Real-time processing, custom consumers | Shard-based (provisioned or on-demand) | Self-managed via KCL |
| Kinesis Data Firehose | 60s – 900s | Data lake ingestion, ETL to storage | Fully automatic | Fully managed |
| Kinesis Data Analytics | Sub-second | Stream transformations, aggregations | Orchestrated by Flink KPU | Managed Apache Flink |
| Kinesis Video Streams | Variable | Media ingestion and playback | Automatic | SDK-based |
Technical internals of Kinesis Data Streams
Kinesis Data Streams operates on a shard-based architecture where each shard represents a fixed unit of capacity. In provisioned mode, a single shard supports 1 MB per second or 1,000 records per second for writes, and 2 MB per second for reads. The shard serves as both the scaling unit and the ordering boundary, making shard design decisions critical for both throughput and data consistency. Records within a shard maintain strict ordering based on sequence numbers assigned at ingestion time. Records across different shards have no ordering guarantees relative to each other.
Partition keys and data distribution
Every record written to Kinesis Data Streams must include a partition key, which the service hashes using MD5 to determine shard placement. The partition key strategy directly impacts both data distribution and ordering semantics. Using high-cardinality partition keys such as user IDs or device identifiers distributes load evenly across shards but sacrifices ordering guarantees for related events. Using low-cardinality keys such as region codes or event types preserves ordering for related records but risks creating hot shards that throttle writes.
The partition key hash maps to a 128-bit integer space, and each shard owns a contiguous range within this space. When you split a shard, the hash range divides into two child shards. When you merge shards, two adjacent ranges combine. This mechanism has direct implications for ordering guarantees during resharding operations, a topic frequently overlooked in competitor documentation.
Enhanced fan-out for dedicated throughput
Standard Kinesis consumers share the 2 MB per second read capacity of each shard using a pull-based GetRecords API. When multiple consumers read from the same stream, they compete for this shared bandwidth, potentially introducing latency spikes during high-throughput periods. Enhanced fan-out addresses this limitation by providing each registered consumer with a dedicated 2 MB per second pipe per shard using HTTP/2 push delivery. This architecture eliminates consumer contention and reduces typical latency from 200 milliseconds to approximately 70 milliseconds.
The trade-off is cost. Enhanced fan-out consumers incur additional charges per consumer-shard-hour and per GB retrieved. For architectures with fewer than three consumers per stream, standard polling often provides sufficient performance at lower cost. For architectures requiring five or more independent consumers, or where latency SLAs demand sub-100ms delivery, enhanced fan-out becomes essential. The following diagram illustrates the difference between standard and enhanced fan-out consumer architectures.
Capacity modes
Kinesis Data Streams offers two capacity modes that represent fundamentally different operational philosophies. Provisioned mode requires you to specify shard count explicitly, giving you precise control over costs and capacity but demanding accurate forecasting and manual scaling operations. On-demand mode automatically scales throughput based on observed traffic patterns, eliminating capacity planning but introducing variable costs that can surprise teams unfamiliar with their traffic profiles.
Provisioned mode characteristics
In provisioned mode, you pay a fixed hourly rate per shard regardless of actual utilization. This model rewards accurate capacity planning with predictable costs. The scaling operations (UpdateShardCount for automatic resharding or manual split and merge operations) take time to propagate and temporarily impact ordering guarantees. Production deployments typically maintain 20-30% headroom above peak throughput to absorb traffic spikes without throttling.
- Write capacity: 1 MB/s or 1,000 records/s per shard, whichever limit is reached first
- Read capacity: 2 MB/s per shard shared across standard consumers, or 2 MB/s per consumer with enhanced fan-out
- Scaling latency: Resharding operations complete within minutes but require consumer coordination
On-demand mode characteristics
On-demand mode, significantly enhanced in 2025, automatically accommodates up to double your previous peak write throughput within minutes. The service maintains this capacity for 15 minutes after traffic decreases, preventing rapid scaling oscillations. You pay per GB ingested and retrieved rather than per shard-hour, making costs directly proportional to actual usage. For workloads with unpredictable traffic patterns or significant daily variation, on-demand mode often reduces both operational burden and total cost.
Recent feature updates in 2025-2026
The past eighteen months have delivered substantial capability improvements to Kinesis Data Streams that directly impact architectural decisions. These updates address long-standing limitations around record size, network configuration, client library efficiency, and data retention that previously pushed some workloads toward alternative solutions.
Large record support up to 10MB
Kinesis Data Streams now supports individual records up to 10 MB, a tenfold increase from the previous 1 MB limit. This enhancement eliminates the need for application-level chunking and reassembly for workloads involving large JSON documents, media metadata, or batch event payloads. The throughput limits remain unchanged at 1 MB per second per shard. A single 10 MB record consumes 10 seconds of write capacity on a single shard. Architects must account for this when designing partition key strategies for large-record workloads.
IPv6 support and dual-stack endpoints
Kinesis Data Streams now supports IPv6 connectivity through dual-stack endpoints, enabling deployments in IPv6-only VPC configurations and simplifying network architecture for organizations transitioning away from IPv4. The dual-stack endpoints automatically negotiate the appropriate protocol based on client configuration, requiring no application code changes for existing deployments.
Extended data retention to 365 days
Data retention options now extend to 365 days, up from the previous maximum of 7 days for standard retention. This capability supports compliance requirements, enables replay of historical data for model retraining, and provides extended recovery windows for consumer failures. Retention costs scale linearly with duration, so architects should implement lifecycle policies that balance retention requirements against storage costs.
KCL 3.x migration and deprecated client libraries
The Kinesis Client Library version 3.x represents a significant architectural evolution from previous versions, introducing improved load balancing, reduced DynamoDB costs for lease management, and better handling of resharding operations. AWS has announced end-of-support timelines for KCL 1.x and KPL 0.x, making migration planning essential for production workloads.
Key improvements in KCL 3.x
KCL 3.x implements a new lease assignment algorithm that reduces DynamoDB read and write operations by up to 50% compared to KCL 2.x, directly lowering operational costs for high-shard-count streams. The library also introduces graceful lease handoff during worker scaling, eliminating the lease-stealing behavior that caused duplicate processing in earlier versions. Integration with the Apache Flink Kinesis connector 5.0.0 provides seamless interoperability for hybrid processing architectures.
The migration path from KCL 1.x requires careful attention to interface changes. KCL 3.x uses a different record processor interface and checkpoint semantics. The following checklist outlines the critical migration steps.
- Audit existing KCL 1.x implementations for deprecated API usage, particularly IRecordProcessor interface implementations
- Update to KCL 2.x as an intermediate step if direct migration to 3.x presents compatibility challenges
- Implement the new ShardRecordProcessor interface with updated lifecycle methods
- Test checkpoint behavior thoroughly, as KCL 3.x uses different DynamoDB table schemas
- Deploy with parallel consumer groups initially to validate processing parity before decommissioning legacy consumers
Ordering guarantees during shard resharding
Maintaining record ordering during shard split and merge operations is one of the most misunderstood aspects of Kinesis Data Streams architecture. The service guarantees ordering within a shard based on sequence numbers. However, resharding operations create transitional states that require careful consumer coordination to preserve end-to-end ordering semantics.
When a shard splits, the parent shard stops accepting new records and enters a CLOSED state. Records continue to be readable from the parent shard until consumers reach the SHARD_END marker. Meanwhile, new records with partition keys that previously mapped to the parent shard now route to one of two child shards based on the new hash range boundaries. A consumer that begins reading from child shards before exhausting the parent shard will observe out-of-order records for partition keys that span the transition.
KCL handles this coordination automatically by tracking shard lineage in its lease table and ensuring parent shards complete before child shard processing begins. Custom consumer implementations must implement equivalent logic by querying the DescribeStream API to discover shard hierarchy and managing checkpoint state accordingly.
Cost optimization and monitoring strategies
Kinesis Data Streams costs accumulate across multiple dimensions. These include shard-hours for provisioned mode, data ingestion and retrieval for on-demand mode, extended retention, and enhanced fan-out consumer fees. Effective cost management requires understanding these components and implementing monitoring that provides visibility into utilization patterns.
Key CloudWatch metrics for capacity planning
The CloudWatch metrics for Kinesis Data Streams provide essential signals for capacity planning and cost optimization. IncomingBytes and IncomingRecords indicate write throughput utilization against shard limits. GetRecords.IteratorAgeMilliseconds reveals consumer lag, with values exceeding your latency SLA indicating under-provisioned consumers or processing bottlenecks. WriteProvisionedThroughputExceeded signals throttling events that may require additional shards or partition key redistribution.
- IncomingBytes/IncomingRecords: Monitor against shard limits to predict scaling needs
- GetRecords.IteratorAgeMilliseconds: Alert when consumer lag exceeds acceptable thresholds
- ReadProvisionedThroughputExceeded: Indicates consumer contention requiring enhanced fan-out
- WriteProvisionedThroughputExceeded: Signals the need for additional shards or partition key optimization
Security and compliance considerations
Kinesis Data Streams integrates with AWS security services to provide encryption, access control, and audit capabilities required for regulated workloads. Server-side encryption using AWS Key Management Service protects data at rest with customer-managed or AWS-managed keys. In-transit encryption uses TLS 1.2 for all API communications. IAM policies control producer and consumer access at the stream and action level, enabling least-privilege configurations.
For compliance-sensitive deployments, Kinesis Data Streams supports VPC endpoints through AWS PrivateLink, eliminating data transit over the public internet. CloudTrail integration provides audit logs for all control plane operations. Data plane logging captures PutRecord and GetRecords API calls for forensic analysis. The extended 365-day retention capability supports regulatory requirements for data preservation without requiring separate archival infrastructure.
Conclusion
AWS Kinesis has matured into a comprehensive streaming platform capable of supporting the most demanding real-time data architectures. The 2025-2026 enhancements, particularly large record support, KCL 3.x efficiency improvements, and extended retention options, address limitations that previously required workarounds or alternative solutions. Architects must now prioritize KCL migration planning given the approaching end-of-support deadlines for legacy client libraries.
The distinction between Kinesis Data Streams and Firehose remains the most critical architectural decision. Choose Streams when latency requirements demand sub-second processing or when custom consumer logic requires fine-grained control. Choose Firehose when the destination is AWS storage services and operational simplicity outweighs latency constraints. Enhanced fan-out transforms the economics of multi-consumer architectures, making dedicated throughput cost-effective for scenarios that previously required complex consumer coordination.
As streaming architectures become foundational to modern applications, the depth of your Kinesis knowledge directly impacts both system reliability and interview performance. Master the shard resharding lifecycle, understand the trade-offs between capacity modes, and build operational runbooks around CloudWatch metrics. These capabilities distinguish engineers who can design streaming systems from those who merely configure them.
