Handling Multipart & Streaming Uploads

Why Buffering the Whole Body Was Rejected

File upload bugs usually start with an API framework making the easy path feel reasonable. In Gin, c.FormFile is perfectly convenient for small files. It is also the wrong default for a production attachment system if it encourages buffering the entire multipart body before the application has even decided whether the upload is allowed. Fieldwork rejected that path because attachments belong to tenants with quotas, projects with access rules, and storage backends with their own latency. A 200 MB body arriving over a slow connection should not become a 200 MB surprise in process memory just because the handler waited too long to start streaming.

The real design question was not how do we parse multipart? It was where do bytes live while policy checks, checksuming, and storage writes happen? The answer we wanted was: mostly in the network reader and the object-store writer, not in our heap and not in temporary local files. That is why this lesson is about streaming, not simply about accepting file uploads.

Streaming Parts Instead of Materializing Files

Fieldwork's upload endpoint uses MultipartReader directly. The handler authenticates the tenant, checks that the user can attach files to the target task, verifies declared content rules, then consumes the file part as a stream. The storage layer accepts an io.Reader, not a byte slice and not a filesystem path. That signature is the actual architectural choice. Once storage accepts a stream, the rest of the stack naturally stops inventing intermediate copies. We rejected helpers that wanted a []byte because they silently pull policy and capacity decisions in the wrong direction.

One subtle decision here is that the handler does not try to parse every form field first and then begin uploading. It processes the stream in order. If metadata matters for storage placement, clients send it before the file part or use a separate metadata creation step. We rejected parse everything, then upload because it assumes the body is cheap to hold while we think. It is not. Large bodies punish indecision.

Validation, Limits, and Failure Modes

Streaming does not mean trusting the stream. Fieldwork still enforces tenant-level size limits, per-content-type allowlists, attachment count limits per task, and optional checksums. The difference is where those validations happen. Some can happen before reading bytes at all, such as tenant quota or task permission. Others happen while bytes move, such as content-length ceilings, MIME sniffing, or hash calculation with io.TeeReader. We rejected the idea that validation requires full buffering. Most useful validation can be done incrementally if the storage path is designed for it.

• Validate permissions and quota before accepting significant upload work.
• Validate size and digest incrementally while streaming.
• Fail fast on unsupported content types instead of finishing an upload you plan to delete.

Metadata Is a Separate Write

The uploaded object and the database record are related but not identical events. Fieldwork writes attachment metadata to Postgres only after storage reports success. That metadata includes tenant ID, task ID, object key, original filename, content type, size, and checksum. We rejected writing the row first and hoping cleanup catches failed uploads later because it creates broken attachment references during the common failure mode. The storage write is the expensive part; until it succeeds, there is no attachment to reference. If the metadata insert fails afterward, we schedule object cleanup as compensating work.

The Decision

Fieldwork handles multipart uploads as streams. The handler authorizes early, reads multipart parts incrementally, validates size and content while bytes move, and writes attachment metadata only after object storage confirms success. We rejected whole-body buffering and temp-file staging because they make server capacity a side effect of client file size. The core decision is simple: attachment bytes are not application data we need in memory. They are a stream to be controlled, validated, and handed off with as little intermediate state as possible.