Records are mutable or immutable data objects (depending on the stream template) stored in a stream. Records are created by ingesting data into a stream.
Records are shaped similarly to instances in the Data Modeling service, and their schema is defined by the containers referenced as sources.
Each record is associated with a space. Space-based access control applies to records.
This section specifies the records resource and uses the following Data Modeling concepts:
Note: Every record has a required top-level externalId property, which can be used
in some queries to retrieve or aggregate records (for filtering or sorting).
For mutable records, externalId (together with space) uniquely identifies a record for write operations (create, upsert, delete).
For immutable records, the records API does not enforce uniqueness of the space + externalId pair, so multiple records may share the same identifier.
Both the rate of requests (denoted as requests per second, or ‘rps’) and the number of concurrent (parallel)
requests are governed by limits for all CDF API endpoints. If a request exceeds a limit,
it will be throttled with a 429: Too Many Requests response. See Resource throttling for limit types and how to avoid throttling.
Limits apply to the API endpoints for this service. Some request types consume more resources (compute, storage I/O) than others. For example, ingest requests are less resource-intensive than analytical requests (Aggregate, Retrieve).
Limits for query endpoints (Sync, Retrieve, Aggregate) are hierarchical:
The Sync endpoint only checks the Query request budget. The Retrieve and Aggregate endpoints must pass both their dedicated budget check AND the Query request budget check.
For example, with mutable streams you can make up to 40 rps total across all query endpoints (Query budget limit), but only up to 20 of those can be Retrieve requests (Retrieve budget limit) and only up to 15 can be Aggregate requests (Aggregate budget limit). That means you could send 20 Retrieve + 15 Aggregate + 5 Sync = 40 total RPS.
Query performance and rate limits vary between mutable and immutable streams because of their different storage characteristics.
Mutable streams provide consistent high-performance queries and higher rate limits (see "mutable streams" limits in the tables below).
Immutable streams are optimized for ingesting very large amounts of data, which results in lower query performance and stricter rate limits than mutable streams (see "immutable streams" limits in the tables below).
When designing data access patterns, use mutable streams for high-performance queries and higher rate limits, or immutable streams when the priority is high-volume ingestion and long-term storage.
The amount of data returned in responses from query endpoints (Sync, Retrieve, Aggregate)
is also limited. Prefer reading only the data you need: use an appropriate filter and limit which sources are retrieved, rather than
retrieving large result sets that you will not use.
The version limits for the records endpoints are shown in the tables below. These limits are subject to change, pending review of consumption patterns and resource availability over time:
| Ingest request budget | ||
|---|---|---|
| Overall | Per ID | |
| Requests per second | 40 | 30 |
| Concurrent requests | 20 | 15 |
| Query request budget | ||
|---|---|---|
| Overall | Per ID | |
| Requests per second: mutable streams | 40 | 30 |
| Concurrent requests: mutable streams | 30 | 22 |
| Requests per second: immutable streams | 10 | 7 |
| Concurrent requests: immutable streams | 10 | 7 |
| Response MB per second | 4 | 3 |
Additional dedicated budgets for Retrieve and Aggregate endpoints:
The Retrieve and Aggregate endpoints have dedicated budgets that are checked in addition to the Query request budget shown above. A request to these endpoints must pass both budget checks.
| Retrieve request budget | ||
|---|---|---|
| Overall | Per ID | |
| Requests per second: mutable streams | 20 | 15 |
| Concurrent requests: mutable streams | 20 | 15 |
| Requests per second: immutable streams | 10 | 7 |
| Concurrent requests: immutable streams | 10 | 7 |
Note: Retrieve endpoint requests are checked against both this budget and the Query request budget above.
| Aggregate request budget | ||
|---|---|---|
| Overall | Per ID | |
| Requests per second: mutable streams | 15 | 12 |
| Concurrent requests: mutable streams | 10 | 7 |
| Requests per second: immutable streams | 5 | 4 |
| Concurrent requests: immutable streams | 5 | 4 |
Note: Aggregate endpoint requests are checked against both this budget and the Query request budget above.
Summary: