Skip to content

Add async iterable support for paginating query, scan, and collection results #563

Description

@anatolzak

Summary

The result of .go() on query, scan, and collection operations should support for await...of iteration, yielding one page of results at a time. This enables streaming-style pagination without manual cursor management.

Motivation

Currently, consuming all pages of a query requires either:

  1. Setting { pages: "all" } and waiting for all pages to accumulate into a single response, or
  2. Manually looping with cursors:
let cursor = null;
do {
  const results = await entity.query.myIndex({ pk }).go({ cursor });
  // process results.data
  cursor = results.cursor;
} while (cursor !== null);

Neither approach gives you per-page control with a clean API. The cursor loop is verbose, and { pages: "all" } forces you to wait for all data before processing any of it.

Proposed API

The result of .go() becomes dual-consumable — it works as a Promise (existing behavior, unchanged) and as an AsyncIterable (new):

// Existing behavior — unchanged
const result = await entity.query.myIndex({ pk }).go({ pages: "all" });
console.log(result.data);

// New — iterate page by page
for await (const page of entity.query.myIndex({ pk }).go({ pages: "all" })) {
  console.log(page.data);   // items for this page
  console.log(page.cursor);  // cursor for this page
}

// Works with scans
for await (const page of entity.scan.go({ pages: "all" })) {
  console.log(page.data);
}

// Works with collections
for await (const page of service.collections.myCollection({ id }).go({ pages: "all" })) {
  console.log(page.data.entityA);
  console.log(page.data.entityB);
}

// Early termination
for await (const page of entity.scan.go({ pages: "all" })) {
  items.push(...page.data);
  if (items.length >= 100) break;
}

Key behaviors

  • No breaking changesawait .go() continues to work exactly as before
  • Lazy — the underlying query is not executed until .then() or for await is called
  • Per-page results — each yielded value has { data, cursor } (and unprocessed when hydrate is used)
  • All execution options respectedlimit, pages, count, data, attributes, etc. all work with iteration
  • Early break supported — breaking out of the loop stops further DynamoDB requests

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions