Skip to content

fix(#433): stream multipart uploads without buffering files#505

Open
ManuelReschke wants to merge 1 commit into
imroc:masterfrom
ManuelReschke:master
Open

fix(#433): stream multipart uploads without buffering files#505
ManuelReschke wants to merge 1 commit into
imroc:masterfrom
ManuelReschke:master

Conversation

@ManuelReschke

Copy link
Copy Markdown

This change makes multipart uploads stream directly to the HTTP transport instead of building the complete multipart body in a bytes.Buffer first.

  • Stream multipart bodies through io.Pipe, keeping memory usage bounded for large uploads.
  • Preserve an exact Content-Length for SetFile and SetFileBytes by counting multipart framing bytes and adding the known file sizes without reading the file contents into memory.
  • Fall back to an unknown content length for sources whose size is not known, while still streaming their contents.
  • Create a fresh multipart stream for each request attempt so file uploads remain replayable for redirects and retries.
  • Reopen files created with SetFile for each attempt and inspect only the first 512 bytes when detecting their content type.
  • Mark SetFileReader bodies as non-replayable, preventing retries from silently sending an exhausted reader.
  • Propagate multipart field, file, and pipe write errors to the HTTP transport instead of discarding them.

Content-Length behavior

  • SetFile and non-empty SetFileBytes uploads retain an exact Content-Length.
  • Custom uploads with both FileSize and ContentType set also retain an exact Content-Length.
  • Uploads with an unknown size use an unknown content length and are streamed without buffering.
  • Explicit force-chunked uploads keep their existing unknown-length behavior.

Validation performed:

go test ./...
go vet ./...
git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant