DynamoDB Streams: Fan-Out, CDC, and Projections
DynamoDB Streams is the single most underused feature on the platform. Most teams turn it on for backup-style use cases (“write to S3 just in case”) and never come back to it. Used well, Streams turn DynamoDB from a database into the source of truth that drives everything downstream - search indexes, materialized views, analytics warehouses, notifications, and event-sourcing read models.
The three patterns below cover 95% of what you’ll ever do with Streams. They compose. The same Stream can drive all three at once.
What Streams actually give you
Every write to a Stream-enabled table appends a record to an ordered log. The record contains:
- The event type (
INSERT,MODIFY,REMOVE) - The old image (before the write) and the new image (after) - configurable
- The partition key and sort key of the affected item
- A sequence number you can use to detect ordering / replays
- A
userIdentityblock (you’ll use this to distinguish TTL deletes from user-initiated deletes)
Records are retained for 24 hours. Within a partition, ordering is preserved. Across partitions, you can’t assume order.
There are two main consumers:
- Lambda triggers (the easy path). AWS Lambda subscribes to the stream and invokes your function with batches of records. This is what you’ll use most of the time.
- Kinesis Data Streams adapter / EventBridge Pipes. For higher throughput, longer retention, or fan-out to non-Lambda consumers. EventBridge Pipes in particular is the modern way to wire Streams to other AWS services without writing transform code.
Pattern 1: Fan-out to multiple consumers
The most common shape. One write to DynamoDB, multiple downstream side effects: send a notification, update a search index, increment an analytics counter, post to a Slack webhook.
The naive way is to do all of that in the request handler that wrote the item. Five service calls, sequential, blocking the user. Any of them fails and you have to decide whether to roll back the DynamoDB write.
The Stream-based approach: write to DynamoDB once. A Lambda picks up the Stream record and fans it out.
// Lambda triggered by DynamoDB Stream
export async function handler(event: DynamoDBStreamEvent) {
for (const record of event.Records) {
if (record.eventName !== "INSERT") continue
const item = unmarshall(record.dynamodb.NewImage)
if (!item.pk?.startsWith("ARTICLE#")) continue
// Fan out in parallel - one slow downstream doesn't block the others
await Promise.allSettled([
indexInOpenSearch(item),
notifySubscribers(item),
incrementCounter("articles_published"),
postToSlack(item),
])
}
}This works because the user-facing write is a single round trip to DynamoDB - no waiting for downstream services. Downstream failures are isolated: if Slack is down, the search index still updates. Lambda retries failed batches automatically, giving you at-least-once delivery without writing your own retry loop. And if downstreams are slow, the Stream just buffers up to its 24-hour limit.
A few things to watch. Lambda can deliver the same record multiple times, so each downstream operation must be safe to apply twice (see idempotency keys). Promise.allSettled lets you log failures without failing the whole batch; if one downstream is consistently failing, dead-letter queue it. If you only care about a subset of items, filter early and return - don’t pay the unmarshalling cost on records you don’t need. Lambda also supports event source filters that prevent your function from being invoked at all for irrelevant records.
Pattern 2: change data capture (CDC) to another database
DynamoDB is good at point lookups and tightly-scoped queries. It’s bad at ad-hoc filtering, full-text search, and analytical aggregations. The standard fix is to project your DynamoDB writes into a database that’s good at those things - OpenSearch for search, Postgres for reporting, Snowflake for warehousing, Redis for fast read caches.
export async function handler(event: DynamoDBStreamEvent) {
for (const record of event.Records) {
const pk = record.dynamodb.Keys.pk.S
if (!pk?.startsWith("PRODUCT#")) continue
if (record.eventName === "REMOVE") {
const oldImage = unmarshall(record.dynamodb.OldImage)
await openSearch.delete({ index: "products", id: oldImage.productId })
continue
}
const newImage = unmarshall(record.dynamodb.NewImage)
await openSearch.index({
index: "products",
id: newImage.productId,
document: {
productId: newImage.productId,
title: newImage.title,
description: newImage.description,
price: newImage.price,
tags: newImage.tags,
},
})
}
}DynamoDB stays the source of truth. OpenSearch is a derived view that gets eventually-consistent updates. You query DynamoDB for “give me product by ID” and OpenSearch for “find products matching this query.”
This is the same shape as fan-out, narrower in scope. It’s also the ideal job for EventBridge Pipes - you can wire DynamoDB Streams → Pipes → OpenSearch without writing a Lambda at all, just a transform expression in the Pipe definition.
Three things to keep in mind here. First, schema drift: if you change the DynamoDB schema, the projection has to adapt - version your projection code and don’t assume old records have the same shape. Second, initial backfill: Streams only see writes after they’re enabled, so you need a one-time backfill (Scan + write) to populate the downstream from existing data before the Stream takes over. Third, eventual consistency: the downstream lags DynamoDB by seconds to minutes, so don’t read your own writes from the projection.
Pattern 3: Read-model projections (event sourcing)
The most powerful pattern, and the most underused. If you store events in DynamoDB - say, every change to an order’s state - you can build any number of read-side views by projecting the Stream into shaped read tables.
EventTable → StreamLambda → OrderSummary table (one row per order)
(append-only) → CustomerStats table (aggregates)
→ PendingOrdersIndex (sparse)Each event in the Stream gets routed to one or more projection updaters. The projections themselves can live in the same DynamoDB table (as separate item types in your single-table design) or in a different table optimized for the read shape.
// Projection updater for OrderSummary
export async function handler(event: DynamoDBStreamEvent) {
for (const record of event.Records) {
if (record.eventName !== "INSERT") continue
const evt = unmarshall(record.dynamodb.NewImage)
if (!evt.pk?.startsWith("ORDER#") || !evt.sk?.startsWith("EVENT#")) continue
switch (evt.eventType) {
case "OrderPlaced":
await OrderSummary.create({ orderId: evt.aggregateId, status: "pending", total: evt.payload.total }).go()
break
case "OrderShipped":
await OrderSummary.update({ orderId: evt.aggregateId })
.set({ status: "shipped", shippedAt: evt.timestamp })
.go()
break
case "OrderCancelled":
await OrderSummary.update({ orderId: evt.aggregateId })
.set({ status: "cancelled", cancelledAt: evt.timestamp })
.go()
break
}
}
}The event log is immutable. The projections are derivable. If you ever want a new read shape, write a new projection that reads the event log from the beginning and builds it. This is what makes event sourcing flexible - you’re not locked into the queries you imagined upfront.
The event sourcing pattern walks through this in full detail, including projection rebuilds and snapshot strategies.
Composing all three
These patterns aren’t exclusive. A single Stream subscriber can do fan-out, CDC, and projections all at once - or you can run multiple subscribers, each with a focused job.
The latter is usually cleaner. Lambda functions per concern:
index-search-projection- syncs to OpenSearchnotification-fanout- sends emails / push / Slackanalytics-counters- increments metrics in a side tableevent-sourcing-projector- builds read models from event records
Each one filters the Stream at the consumer (or via Lambda event source filters) and ignores records it doesn’t care about. Failures in one don’t affect the others. You can deploy and scale them independently.
Stream choices
DynamoDB lets you configure what each record contains:
- Keys only: just PK and SK. Cheapest, but you can’t see what changed without re-reading the item.
- New image: full item after the write. Good for CDC and projections.
- Old image: full item before the write. Useful for diffing, audit logs.
- New and old images: both. Most expressive, most expensive.
Pick based on what your consumers need. “New and old” is the safe default; “keys only” is the right call when consumers will re-fetch the item anyway.
When to use Kinesis instead
DynamoDB Streams have a 24-hour retention window. Kinesis Data Streams (via the Kinesis Data Streams adapter for DynamoDB) gives you up to a year. You’d want this if:
- Consumers can fall behind for more than 24 hours and need to catch up
- You want to replay history into a new projection (rebuild a read model from scratch)
- You have many parallel consumers and Stream’s per-shard throughput limits become a problem
For most use cases, the standard Streams + Lambda combo is fine. Move to Kinesis only when you’ve hit a real limit.
Common bugs
Streams are not exactly-once. Lambda batches can be retried; the same record can be delivered multiple times. Make all consumer side effects idempotent.
Don’t read the projection in the same request that wrote the source. The downstream lags DynamoDB - don’t write to DynamoDB and then immediately query OpenSearch, you’ll race.
Watch the consumer lag. If your Lambda is slow or failing, records pile up and eventually drop off the 24-hour retention. Monitor the iterator age metric. Anything close to 24h is an outage.
Handle REMOVE events explicitly. A REMOVE event has OldImage only, not NewImage. Code that assumes NewImage exists will throw on TTL deletes (see TTL patterns).
Don’t assume cross-partition ordering. Within one partition key, Stream order matches write order. Across partitions, there’s no ordering guarantee - don’t assume “all events for this user came in this order” if events are partitioned by something other than user.
Mental model
Treat the Stream as the spinal cord of your system. DynamoDB writes are the source of truth - everything else is a derivation that the Stream keeps in sync. Search indexes, analytics, notifications, projections - they’re all just functions of the Stream over time.
Once you’re thinking this way, you stop writing application code that does five things at once and start writing code that does one thing (write to DynamoDB) and trusts the Stream to handle the rest.
Streams are how a single-table DynamoDB design scales to a real system. Patterns at singletable.dev - especially event sourcing, analytics events, and IoT time-series - all assume Streams as the integration point. I’m building singletable.dev to make those integration points part of the schema, not an afterthought.
