← Integration Engineering
Integration Engineering

Syncing Data: Copying Once Is Easy, Keeping It in Sync Is the Job

8 Oct 2026·3 min read

A copy is a photo. The moment you take it, the real data starts moving again. Keeping two systems matching every day is called syncing, and it is a different job.

Full sync once, then only what changed

Start by copying every record once. That is the full sync. After that, ask only for records changed since the last run, using a field like updated_at. That is an incremental sync, and it stays fast as the data grows.

curl -s "https://api.github.com/repos/octocat/Hello-World/issues?since=2026-09-01T00:00:00Z&state=all&per_page=2"
A real incremental request. GitHub’s since returns only results updated after that time. I ran it without a token. Your results will differ.

Save where you stopped

Store the point where the last run stopped, called a checkpoint, in your database. Save it only after the records are safely written. If the job crashes, it continues from there, not from zero.

The hard part: deletes

Many APIs do not tell you when a record is deleted. It stops appearing, and your copy keeps it forever. Look for a deleted flag or a deletes endpoint. If there is neither, compare the ID lists now and then, like once a night.

Do this in 60 secondsOpen the docs of one API you sync from. Search for "since", "updated" or "modified". Then search for "deleted". What you find tells you how hard your sync will be.
Today’s AI angleAn AI agent reading from a synced copy only knows what the sync brought over. If deletes are not handled, it may email someone who asked to be deleted. Stale data becomes wrong actions.
Syncing Data: Copying Once Is Easy, Keeping It in Sync Is the Job · Ali Huzaifa