← Integration Engineering
Integration Engineering

No API Gives You All the Data in One Call

2 Oct 2026·3 min read

A sync is tested with 20 records and everything arrives. Then a real customer connects with 5,000 records, and only the first 100 show up. No error. The code read the first page and stopped. APIs send data in pages, maybe 50 or 100 items at a time, and your code has to ask for the next one.

Two ways to ask for the next page

  • Page number or offset: "give me page 3". Easy, but if rows are added or deleted while you read, you can skip or repeat some.
  • Cursor: the API gives you a marker, and you say "give me what comes after this". More reliable, and APIs like Stripe and Slack use it. Do not read or build the cursor yourself; pass it back as it is.
curl -I "https://api.github.com/users/octocat/repos?per_page=2"
Asks GitHub for 2 repos per page and shows only the headers. No login needed. The Link header in the reply has rel="next", the page to ask for next.

The rules that keep you safe

  • Keep asking until the API says there is no next page or next cursor. Do not guess the end from the count.
  • In long syncs, save the last cursor, so a crash can continue from there.
  • Always test with more data than one page. Set a tiny page size, like per_page=2, and a broken loop shows up right away.
Do this in 60 secondsFind one place in your code that fetches a list from an API. Look for where it reads the next page, cursor or Link header. If you cannot find it, you found the bug before your customer did.
Today’s AI angleA "list customers" tool that returns only the first page makes an AI agent say "you have 100 customers". A good tool follows the pages itself, or says "more results available".
No API Gives You All the Data in One Call · Ali Huzaifa