Upload multipart file

Multipart file upload enables upload of files larger than 5 GiB, using a uniform API on all cloud environments for CDF.

Each file part must be larger than 5 MiB, and smaller than 4000 MiB. The file part for the last uploadURL can be smaller than 5 MiB. Maximum 250 upload URLs can be requested. The supported maximum size for each file uploaded with the multi-part upload API is therefore 1000 GiB (1.073 TB / 0.976 TiB). The client should calculate the ideal number of parts depending on predetermined or estimated file size, between 1 and the maximum. Specify the number of parts in the parts URL query parameter. The parts parameter is required.

The request creates metadata information for a new file, and returns in addition to the file id, also a uploadId, and a list of uploadUrls for uploading the file contents. To upload a file, send an HTTP PUT request to each of the uploadUrls, with the corresponding part of the file in the request body. You may use a 'Content-Length' header in the PUT request for each part, but this is not required. A failed part PUT upload can be retried.

The client must ensure that the parts of the source file are stored in the correct order, using the order of the uploadUrls as specified in the response. This to avoid ending up with a corrupt final file.

The parts can optionally be uploaded in parallel, preferably on a subset of parts at a time, for example maximum 3 concurrent PUT operations.

Once all file parts have been uploaded, the client should call the 'files/completemultipartupload' endpoint, with the required file ID (as id or externalId) and uploadId fields in the request body. This will assemble the parts into one file. The file's uploaded flag will then eventually be set to true.

A standard sequence of calls to upload a large file with multipart upload would be for example as follows:

  1. POST files/initmultipartupload?parts=8, to start a multipart upload session with 8 parts. Expect a 201 CREATED response code, and a response body with information to be used in the part uploads and completemultipartupload requests.
  2. PUT uploadUrl, for each of the uploadUrls in the response from files/initmultipartupload. Expect a 200 OK or 201 CREATED response for each PUT request.
  3. POST files/completemultipartupload, with request body '{ "id":123456789, "uploadId":"ABCD4321EFGH" }'. This will assemble the file. Expect a 200 OK response.

Consider verifying that the file is eventually marked as uploaded with a call to the getFileByInternalId endpoint.

NOTE: The uploadUrls expires after one week. A file that does not have the file content parts uploaded and completed within one week will be automatically deleted.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
query Parameters
overwrite
boolean
Default: false

If 'overwrite' is set to true, and the POST body content specifies a 'externalId' field, fields for the file found for externalId can be overwritten. The default setting is false.

If metadata is included in the request body, all of the original metadata will be overwritten. The actual file will be overwritten after a successful upload with the uploadUrls from the response. If there is no successful upload, the current file contents will be kept.

File-Asset mappings only change if explicitly stated in the assetIds field of the POST json body. Do not set assetIds in request body if you want to keep the current file-asset mappings.

parts
required
integer <int32> [ 1 .. 250 ]

The 'parts' parameter specifies how many uploadURLs should be returned, for uploading the file contents in parts. See main endpoint description for more details.

Request Body schema: application/json
required

Fields to be set for the file.

externalId
string (CogniteExternalId) <= 255 characters

The external ID provided by the client. Must be unique for the resource type.

name
required
string (FileName) <= 256 characters

Name of the file.

directory
string (FileDirectory) <= 512 characters

Directory containing the file. Must be an absolute, unix-style path.

source
string (FileSource) <= 128 characters

The source of the file.

mimeType
string (MimeType) <= 256 characters

File type. E.g. text/plain, application/pdf, ..

object (FilesMetadataField)

Custom, application specific metadata. String key -> String value. Limits: Maximum length of key is 128 bytes, value 10240 bytes, up to 256 key-value pairs, of total size at most 10240.

assetIds
Array of integers <int64> (CogniteInternalId) [ 1 .. 1000 ] items
dataSetId
integer <int64> (DataSetId) [ 1 .. 9007199254740991 ]

The dataSet Id for the item.

sourceCreatedTime
integer <int64> >= 0

The timestamp for when the file was originally created in the source system.

sourceModifiedTime
integer <int64> >= 0

The timestamp for when the file was last modified in the source system.

securityCategories
Array of integers <int64> (CogniteInternalId) [ 0 .. 100 ] items

The security category IDs required to access this file.

Array of objects (LabelList) [ 0 .. 10 ] items unique

A list of the labels associated with this resource item.

object (GeoLocation)

Geographic metadata.

Responses
201

The response for a successful request to initiate upload of multiple parts for a file.

400

The response for a failed request.

post/files/initmultipartupload
Request samples
application/json
{
  • "externalId": "my.known.id",
  • "name": "string",
  • "directory": "string",
  • "source": "string",
  • "mimeType": "image/jpeg",
  • "metadata": {
    • "property1": "string",
    • "property2": "string"
    },
  • "assetIds": [
    • 1
    ],
  • "dataSetId": 1,
  • "sourceCreatedTime": 0,
  • "sourceModifiedTime": 0,
  • "securityCategories": [
    • 1
    ],
  • "labels": [
    • {
      }
    ],
  • "geoLocation": {
    • "type": "Feature",
    • "geometry": {
      },
    • "properties": { }
    }
}
Response samples
application/json
{
  • "externalId": "my.known.id",
  • "name": "string",
  • "directory": "string",
  • "source": "string",
  • "mimeType": "image/jpeg",
  • "metadata": {
    • "property1": "string",
    • "property2": "string"
    },
  • "assetIds": [
    • 1
    ],
  • "dataSetId": 1,
  • "sourceCreatedTime": 0,
  • "sourceModifiedTime": 0,
  • "securityCategories": [
    • 1
    ],
  • "labels": [
    • {
      }
    ],
  • "geoLocation": {
    • "type": "Feature",
    • "geometry": {
      },
    • "properties": { }
    },
  • "id": 1,
  • "uploaded": true,
  • "uploadedTime": 0,
  • "createdTime": 0,
  • "lastUpdatedTime": 0,
  • "uploadId": "string",
  • "uploadUrls": [
    • "uploadURL_for_part_1",
    • "uploadURL_for_part_2",
    • "uploadURL_for_part_3"
    ]
}