← Integration Engineering
Integration Engineering

Files Break Integrations That JSON Never Did

13 Oct 2026·3 min read

An integration can move records for months without trouble. Each one is a few kilobytes of JSON. Then a customer attaches a 400 MB video, and the same code runs out of memory, times out, or sends half a file.

Stream, do not load

Do not read the whole file into memory. Stream it: pass it along in small pieces, so memory stays small however big the file is. Most HTTP libraries and cloud storage SDKs support this.

The link is not the file

Many APIs send a temporary download link instead of the file. GitHub’s docs say archive links for private repos expire after five minutes. AWS calls its version a presigned URL and describes it as a bearer token: whoever holds the link can use it.

curl -sI https://api.github.com/repos/octocat/Hello-World/tarball
I ran it while writing this: it returned HTTP/1.1 302 Found and a Location header pointing to codeload.github.com. The API sent a link, not the file.

So use the link right away, never log it or store it, and keep the file ID to ask for a fresh link later.

Limits and access

  • Each provider has its own size and type limits. Check them before any bytes move, and show a clear message like "This file is too big for this tool".
  • File access is often per file or per folder. When a customer says "half our files are missing", often the user who connected does not have access to those folders.
Do this in 60 secondsFind the place where your code downloads a file from an API. Does it read the whole body into memory, or stream? Try it with the biggest file a customer could send.
Today’s AI angleGive an AI agent file metadata first, like name, size and type, not the whole file. And never paste a temporary download link into its context or logs.
Files Break Integrations That JSON Never Did · Ali Huzaifa