Multipart file upload enables upload of files larger than 5 GiB, using a uniform API on all cloud environments for CDF.

Each file part must be larger than 5 MiB, and smaller than 4000 MiB. The file part for the last uploadURL can be smaller than 5 MiB. Maximum 250 upload URLs can be requested. The supported maximum size for each file uploaded with the multi-part upload API is therefore 1000 GiB (1.073 TB / 0.976 TiB). The client should calculate the ideal number of parts depending on predetermined or estimated file size, between 1 and the maximum. Specify the number of parts in the parts URL query parameter. The parts parameter is required.

The request returns in addition to the file id, also a uploadId, and a list of uploadUrls for uploading the file contents. To upload a file, send an HTTP PUT request to each of the uploadUrls, with the corresponding part of the file in the request body. You may use a 'Content-Length' header in the PUT request for each part, but this is not required. A failed part PUT upload can be retried.

The client must ensure that the parts of the source file are stored in the correct order, using the order of the uploadUrls as specified in the response. This to avoid ending up with a corrupt final file.

The parts can optionally be uploaded in parallel, preferably on a subset of parts at a time, for example maximum 3 concurrent PUT operations.

Once all file parts have been uploaded, the client should call the 'files/completemultipartupload' endpoint, with the required file ID (as id or externalId) and uploadId fields in the request body. This will assemble the parts into one file. The file's uploaded flag will then eventually be set to true.

A standard sequence of calls to upload a large file with multipart upload would be for example as follows:

1. POST files/initmultipartupload?parts=8, to start a multipart upload session with 8 parts. Expect a 201 CREATED response code, and a response body with information to be used in the part uploads and completemultipartupload requests.
2. PUT uploadUrl, for each of the uploadUrls in the response from files/initmultipartupload. Expect a 200 OK or 201 CREATED response for each PUT request.
3. POST files/completemultipartupload, with request body '{ "id":123456789, "uploadId":"ABCD4321EFGH" }'. This will assemble the file. Expect a 200 OK response.

Consider verifying that the file is eventually marked as uploaded with a call to the getFileByInternalId endpoint.

NOTE: The uploadUrls expires after one week. A file that does not have the file content parts uploaded and completed within one week will be automatically deleted.

Request throttling

This endpoint is a subject of the new throttling schema (limited request rate and concurrency). Please check Files resource description for more information.