Transformations

Transformations enable users to use Spark SQL queries to transform data from the CDF staging area, RAW, into the CDF data model.

Concurrency limits

The number of concurrent (parallel) jobs are governed by limits. If a request exceeds one of the limits, the job fails to run. This limitation also applies to scheduled transformations.

The limits for the transformation service are described in the table below. Note that under high load, some deviation from the limits might occur for short periods as the service scales up.

Limit Per project
Concurrency 10 parallel jobs

List transformations

Required capabilities: transformationsAcl:READ

List transformations. Use nextCursor to paginate through the results.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
query Parameters
limit
integer <int32> [ 1 .. 1000 ]

Limits the number of results to be returned. The maximum is 1000, default limit is 100.

cursor
string

Cursor for paging through results.

includePublic
boolean

Whether public transformations should be included in the results. The default is true.

withJobDetails
boolean

Whether transformations should contain information about jobs. The default is true.

Responses
200

Paged response with list of transformations.

400

The response for a failed request.

403

The response for a forbidden request.

get/transformations
Response samples
application/json
{
  • "items": [
    ],
  • "nextCursor": "string"
}

Create transformations

Required capabilities: transformationsAcl:WRITE

Create a maximum of 1000 transformations per request.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
Array of objects (TransformationCreate) <= 1000 items
Responses
200

Response with list of transformations.

400

The response for a failed request.

403

The response for a forbidden request.

409

The response for a conflict.

post/transformations
Request samples
application/json
{
  • "items": [
    ]
}
Response samples
application/json
{
  • "items": [
    ]
}

Retrieve transformations

Required capabilities: transformationsAcl:READ

Retrieve a maximum of 1000 transformations by ids and externalIds per request.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
Array of TransformationCogniteExternalId (object) or TransformationCogniteInternalId (object) <= 1000 items
ignoreUnknownIds
boolean

Ignore IDs and external IDs that are not found. Defaults to false.

withJobDetails
required
boolean

Whether the transformations will be returned with running job and last created job details.

Responses
200

Response with list of transformations.

400

The response for a failed request.

403

The response for a forbidden request.

409

The response for a conflict.

post/transformations/byids
Request samples
application/json
{
  • "items": [
    ],
  • "ignoreUnknownIds": true,
  • "withJobDetails": true
}
Response samples
application/json
{
  • "items": [
    ]
}

Filter transformations

Required capabilities: transformationsAcl:READ

Filter transformations. Use nextCursor to paginate through the results.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
required
object (TransformationFilter)
limit
integer <int32>
cursor
string
Responses
200

Paged response with list of transformations.

400

The response for a failed request.

403

The response for a forbidden request.

post/transformations/filter
Request samples
application/json
{
  • "filter": {
    },
  • "limit": 0,
  • "cursor": "string"
}
Response samples
application/json
{
  • "items": [
    ],
  • "nextCursor": "string"
}

Delete transformations

Required capabilities: transformationsAcl:WRITE

Delete a maximum of 1000 transformations by ids and externalIds per request.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
Array of TransformationCogniteExternalId (object) or TransformationCogniteInternalId (object) <= 1000 items
ignoreUnknownIds
boolean

Ignore IDs and external IDs that are not found. Defaults to false.

Responses
200

Empty response.

400

The response for a failed request.

403

The response for a forbidden request.

409

The response for a conflict.

post/transformations/delete
Request samples
application/json
{
  • "items": [
    ],
  • "ignoreUnknownIds": true
}
Response samples
application/json
{ }

Update transformations

Required capabilities: transformationsAcl:WRITE

Update the attributes of transformations, maximum 1000 per request.

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
Array of UpdateItemWithExternalId_TransformationUpdate (object) or UpdateItemWithId_TransformationUpdate (object) <= 1000 items
Responses
200

Response with list of transformations.

400

The response for a failed request.

403

The response for a forbidden request.

409

The response for a conflict.

post/transformations/update
Request samples
application/json
{
  • "items": [
    ]
}
Response samples
application/json
{
  • "items": [
    ]
}

Run a transformation

Required capabilities: transformationsAcl:WRITE

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
One of:
externalId
required
string (CogniteExternalId) <= 255 characters

The external ID provided by the client. Must be unique for the resource type.

object (NonceCredentials)

Credentials for a CDF session (sessionId, nonce, and project name). These are obtained when you create a session via the Sessions API. See Sessions API documentation for more information.

Responses
200

Response with a single job.

400

The response for a failed request.

403

The response for a forbidden request.

post/transformations/run
Request samples
application/json
{
  • "externalId": "my.known.id",
  • "nonce": {
    }
}
Response samples
application/json
{
  • "id": 0,
  • "uuid": "string",
  • "transformationId": 0,
  • "transformationExternalId": "string",
  • "sourceProject": "string",
  • "destinationProject": "string",
  • "destination": {
    },
  • "conflictMode": "abort",
  • "query": "string",
  • "createdTime": 0,
  • "startedTime": 0,
  • "finishedTime": 0,
  • "lastSeenTime": 0,
  • "error": "string",
  • "ignoreNullFields": true,
  • "status": "Completed"
}

Cancel a transformation

Required capabilities: transformationsAcl:WRITE

Securityoidc-token or oauth2-client-credentials or oauth2-open-industrial-data or oauth2-auth-code
Request
Request Body schema: application/json
required
One of:
externalId
required
string (CogniteExternalId) <= 255 characters

The external ID provided by the client. Must be unique for the resource type.

Responses
200

Empty response.

400

The response for a failed request.

403

The response for a forbidden request.

post/transformations/cancel
Request samples
application/json
{
  • "externalId": "my.known.id"
}
Response samples
application/json
{ }
➔ Next to Transformation Jobs