DynamoDB Pagination Done Correctly
DynamoDB pagination looks deceptively simple. The API gives you a LastEvaluatedKey in the response when there are more results, you pass it back as ExclusiveStartKey on the next request, and you keep going until LastEvaluatedKey is missing. Done.
In practice, almost every team I’ve seen ship a DynamoDB API has at least one pagination bug. Empty pages. Skipped items. Tokens that leak the schema. Pages that change size unpredictably. “Total count” features that don’t work. Once you’ve seen the bugs, the right pattern is obvious - but the docs don’t lay it out clearly enough that you’ll write it correctly the first time.
This post is the playbook.
What actually surprises people
Three things about how DynamoDB pagination works that most teams learn the hard way.
LastEvaluatedKey is opaque. It’s a key, but treat it as a black box. Don’t construct it manually, don’t parse it, don’t assume it’s the same as the SK of the last item returned.
An empty Items array doesn’t mean “no more data.” If you have a FilterExpression, DynamoDB reads up to the page limit (1MB or your Limit), then applies the filter. The page might be empty. You still need to follow LastEvaluatedKey until it’s gone.
Page size isn’t what you think it is. Limit: 10 doesn’t mean “return 10 matching items.” It means “stop after evaluating 10 items.” With a filter, you might get 0 items back. Without a filter, you might still get fewer if you hit the 1MB response cap.
Internalize those and you’ve avoided most pagination bugs already.
The basic pattern
async function* paginateAll<T>(query) {
let cursor: Record<string, any> | undefined = undefined
do {
const result = await query.go({ cursor })
yield* result.data
cursor = result.cursor
} while (cursor)
}
// Usage
for await (const order of paginateAll(OrderEntity.query.byCustomer({ customerId }))) {
process(order)
}The do { } while is intentional - even if the first page is empty, you have to check whether more pages follow. ElectroDB’s cursor is the encoded LastEvaluatedKey; raw SDK users get the LEK directly and pass it as ExclusiveStartKey.
If you’re streaming through all results, this is enough. If you’re paginating to a client, you need more.
Empty pages from FilterExpression
This bites everybody once.
// Looks fine. It isn't.
const result = await OrderEntity.query.byCustomer({ customerId })
.where(({ status }, { eq }) => eq(status, "shipped"))
.go({ limit: 10 })
return result.data // might be emptyIf the customer has 1,000 pending orders followed by 1 shipped order, this returns an empty array - DynamoDB read the first 10 (all pending), filtered them out, and stopped. Your client thinks “no shipped orders” when there’s one waiting.
The fix is to always check cursor regardless of data.length:
async function findShippedOrders(customerId: string) {
let cursor: any = undefined
const results: Order[] = []
do {
const page = await OrderEntity.query.byCustomer({ customerId })
.where(({ status }, { eq }) => eq(status, "shipped"))
.go({ cursor, limit: 100 })
results.push(...page.data)
cursor = page.cursor
if (results.length >= TARGET) break
} while (cursor)
return results
}Better yet: avoid FilterExpression for queries you paginate often. Reshape the SK or add a sparse GSI so the filter happens at the index level. This is one of the highest-leverage uses of sparse indexes.
Client-facing API: encode the cursor
If you expose pagination to API clients (web, mobile, third-party), don’t pass LastEvaluatedKey directly. Three reasons:
- It contains your schema. The keys in the LEK are your table’s PK/SK names. Exposing them tells attackers exactly what your indexes look like.
- It contains user data. The values can include user-identifiable IDs, timestamps, etc. Some of those might be sensitive in URL-bar bookmarks.
- It’s not URL-safe. LEK is a JSON object with potentially nested values. URL-encoding works but is ugly.
Encode it:
function encodeCursor(lek: object): string {
return Buffer.from(JSON.stringify(lek)).toString("base64url")
}
function decodeCursor(cursor: string): object {
return JSON.parse(Buffer.from(cursor, "base64url").toString())
}For sensitive data, sign or encrypt the cursor. A signed cursor (HMAC + secret key) prevents clients from forging cursors that point at items they shouldn’t see. An encrypted cursor hides the schema entirely.
ElectroDB does this for you (page.cursor is already a base64-encoded string), which is most of why I use it - the raw SDK forces every team to invent the same pattern.
Page size and over-fetching
If your client wants 20 items per page, don’t request 20 from DynamoDB. Request more, because of the filter problem above.
const PAGE_SIZE = 20
const FETCH_FACTOR = 5 // fetch up to 5x to compensate for filtered items
async function getPage(cursor?: string) {
const matches: Order[] = []
let nextCursor = cursor
while (matches.length < PAGE_SIZE && nextCursor !== null) {
const page = await OrderEntity.query.byCustomer({ customerId })
.where(/* ... */)
.go({ cursor: nextCursor, limit: PAGE_SIZE * FETCH_FACTOR })
matches.push(...page.data)
nextCursor = page.cursor ?? null
}
// Trim to PAGE_SIZE; remember the cursor of the last item if we over-fetched
const trimmed = matches.slice(0, PAGE_SIZE)
// ...
}If you over-fetched and have leftover items, you have a choice: return them in this page (page size becomes a soft cap) or hold them for the next page (which is more complex but gives stable page sizes). Most APIs use the first - “up to N items per page.”
For pages without FilterExpression, this isn’t necessary; Limit: 20 returns up to 20 items reliably.
”Get total count” - usually a mistake
A common ask: “show me page 5 of 23.” That requires knowing the total count.
DynamoDB cannot give you a total count cheaply. Your options are:
Select: COUNT- returns the count without the items, but still scans the entire matched set. For a large query, this is the same cost as fetching all items. Not free.- Maintain a counter - increment on insert, decrement on delete, with conditional writes. Accurate but adds write overhead and complexity.
- Stream-based aggregation - compute counts in a downstream system (OpenSearch, ClickHouse) via DynamoDB Streams. The right answer when you need many such aggregates.
- Don’t show total count - cursor-based pagination (“Next →”) instead of numbered pagination. This is what most modern apps actually do, and it’s the right call most of the time.
If your UI says “page 5 of 23, showing items 81-100”, you’re committing yourself to one of options 1-3. If it says “Next →” / “Previous”, option 4 is free.
Reverse pagination (“Previous” button)
DynamoDB doesn’t give you a “previous page” cursor. The pagination is forward-only.
Three workarounds:
- Stack of cursors on the client - keep an array of cursors as the user pages forward. Going back means reusing a stored cursor. Works for “Previous” but not for jumping to arbitrary pages.
- Reverse the query - run the same query with
ScanIndexForward: falseand the inverse cursor. Awkward, error-prone. - Cache pages - store rendered pages in the client (or in a session cache) and serve “Previous” from cache. Most pragmatic for search-result UIs.
For most app UIs, option 1 is the right answer. Cursor-stack pagination is fine; “jump to page 17” pagination is the design choice you should be questioning.
”Show me items after this one” (real cursor pagination)
If your sort key is meaningful (e.g. ULIDs ordered by time), you can build a cursor from the application data instead of LastEvaluatedKey:
// "Show me orders after this orderId"
const result = await OrderEntity.query
.byCustomer({ customerId })
.gt({ orderId: cursor })
.go({ limit: 20 })This is conceptually identical to LastEvaluatedKey-based pagination but uses application-meaningful values. The advantage: cursors survive schema changes (an opaque LEK might break if you change index keys; an orderId cursor doesn’t). The disadvantage: only works on indexes where the sort key is uniquely identifying, and you can’t directly support filtered queries.
For event-sourced systems, this is the cleanest pattern - “give me events after sequence X” maps directly to SK > X.
Pagination on a GSI
LastEvaluatedKey from a GSI query includes both the GSI’s PK/SK and the base table’s PK/SK. If you encode the cursor and pass it back, you have to decode it for the same index. Don’t try to use a cursor from one index against another - it’ll throw or, worse, return incorrect results.
This is one more reason to encode cursors with index information baked in:
function encodeCursor(lek: object, indexName: string): string {
return Buffer.from(JSON.stringify({ idx: indexName, lek })).toString("base64url")
}Reject the cursor if it doesn’t match the current query’s index.
Edge cases that bite
Concurrent writes during pagination: new items inserted between page fetches might show up out of order or be missed entirely, depending on where they sort. There’s no way around this without a transactional snapshot, which DynamoDB doesn’t provide for queries. For most use cases, accept the inconsistency.
Deleted items: if an item is deleted between page fetches, it’s gone from the next page. Usually fine. If the delete happens during the read of a page, you might see partial data. This is usually invisible to users.
Massive single-partition queries: if a single PK has 10 million items and you paginate through them, each page is fast but the total is slow. Consider whether the access pattern needs all 10 million items - usually not.
Hidden 1MB cap: each Query response caps at 1MB regardless of Limit. With large items (> 100KB), you’ll get fewer items than Limit says. Don’t assume “Limit: 100” means “100 items returned.”
Common bugs in one place
| Bug | Symptom | Fix |
|---|---|---|
Stop on empty Items | Misses results behind a filter | Always check LastEvaluatedKey |
| Pass raw LEK to client | Schema leak, broken if index changes | Encode + sign cursor |
Limit: 20 for 20 items | Sometimes returns 0-19 items | Over-fetch with a multiplier |
| ”Total count” on every page | Slow and expensive | Cursor-based UI or counter entity |
| Reuse cursor on different index | Errors or wrong results | Tag cursor with index name |
| Forget cursor on subsequent pages | Returns the same page forever | Pass ExclusiveStartKey from previous response |
A correct pagination handler
async function listOrders(req: { customerId: string; cursor?: string; limit?: number }) {
const limit = Math.min(req.limit ?? 20, 100)
const cursor = req.cursor ? decodeCursor(req.cursor) : undefined
if (cursor && cursor.idx !== "byCustomer") {
throw new BadRequest("invalid cursor")
}
const result = await OrderEntity.query
.byCustomer({ customerId: req.customerId })
.where(({ status }, { eq }) => eq(status, "active"))
.go({ cursor: cursor?.lek, limit: limit * 5 })
const orders = result.data.slice(0, limit)
const nextCursor =
result.cursor && orders.length === limit
? encodeCursor({ idx: "byCustomer", lek: result.cursor })
: null
return { orders, nextCursor }
}Bounded limit. Decoded and validated cursor. Over-fetch to handle filters. Encoded next cursor. Null cursor when no more data. This is the shape every paginated DynamoDB endpoint should have.
Mental model
Pagination in DynamoDB is forward-only, opaque, and best-effort consistent. The state lives in the cursor and nothing else. Every bug I’ve seen comes from one of three things: assuming the cursor encodes more than it does, treating an empty page as the end of the data, or exposing the cursor in a form the client can break.
Get the basic shape right (always check the cursor, encode it on the way out, validate it on the way in) and pagination is one of the simpler problems in your DynamoDB layer. Get it wrong and it’s a recurring source of subtle bugs.
Most of the patterns in the pattern library are designed for cursor-based pagination - the social media feed, chat messaging, and analytics events all rely on it. Numbered pagination is rare for a reason.
