Hosted rollups

Ingesting events directly via the Orb API comfortably supports well into the thousands of events per second, still allowing for brief spikes in event volume, and covers the vast majority of use cases, including infrastructure customers. Orb stores your raw event data, which gives you the most flexibility because you can query it in any way you’d like with billable metrics.

Once you’re sustaining more than roughly 10,000 events per second, we recommend moving to a cloud storage bucket sync (S3 or GCS) as your integration path, rather than sending events directly to the API. Batching events into files that sync to a bucket meaningfully improves the resiliency of your integration at high volumes, and it’s the approach we suggest once you’re consistently operating at this scale. This is guidance rather than a hard requirement, and is a separate ingestion option from hosted rollups described below—it does not require purchasing or enabling hosted rollups.

Once you’re operating at a much larger scale—generally north of 500,000 events per second—and want Orb to pre-aggregate that data in real time rather than storing every raw event, Orb offers a service called “hosted rollups,” rolling up your event data as it’s ingested into Orb. These rollups ensure fast access to real-time usage data spanning petabytes of events.

Orb’s hosted rollups architecture has undergone sustained stress testing well beyond 500,000 events per second, and Orb’s largest customers use hosted rollups to sustain several million events per second in production. If you have a specific event load you’re looking to support, our team can help you set up a test environment to illustrate how Orb can support your volume.

Architecture and configuration

Orb’s hosted rollups typically uses an intermediate cloud bucket as a durable message queue for events. A rollup configuration in Orb is defined by:

A set of grouping properties. This is a tuple of properties that identify each grouping that Orb emits per timeframe.
An aggregation function for each property you’d like to aggregate in a given rollup. By default, Orb will also include the count of events in the rollup.
The time window over which you’d like to compute the final rollups (default is 10 minutes).
The frequency at which you’d like Orb to emit partial rollups (default is 30 seconds).

Orb emits partial frequent rollups to ensure you have access to usage data as it arrives, not just when the aggregation window is complete.

Example: Rollups on data download

Imagine you’re a file storage company that charges by the number of files downloaded and the total bytes downloaded. You offer three tiers of download speeds (fast, medium, and slow), charging more for higher speeds. Your customers download hundreds of thousands of files every second, and you send every download to Orb as a usage event with the following shape:

{
  "id": "afdb028a-c510-11ed-9faa-0a58a9feac02",
  "customer": "617cb4de-c511-11ed-9faa-0a58a9feac02",
  "download_timestamp": "2022-08-14T15:14:31.132Z",
  "bytes_downloaded": 3256,
  "download_speed": "fast"
}

You configure Orb’s hosted rollups with the following properties:

id is the field Orb will use to deduplicate events.
The customer property maps to external_customer_id, and uniquely identifies an Orb customer.
download_timestamp is the timestamp used to bucket events into rollup windows
Sum over bytes_downloaded grouping by download_speed
Aggregate over 10 minute windows, emitting partial rollups every 30 seconds

Orb will then ingest these aggregates with timestamp as the start of the aggregation window:

{
  "idempotency_key": <ORB_GENERATED_IDEMPOTENCY_KEY>,
  "timestamp": "2022-08-14T15:10:00.000Z",
  "external_customer_id": "617cb4de-c511-11ed-9faa-0a58a9feac02",
  "count": 3037816,
  "sum_bytes_downloaded": 25343235423,
  "download_speed": "fast"
}

You’ll note that bytes_downloaded in the original payload translates to sum_bytes_downloaded in the ingested event, signifying that Orb has automatically summed this field for all events that have download_speed: fast, in the 10 minute timeframe 2022-08-14T15:10:00.000Z to 2022-08-14T15:``20``:00.000Z. Once these events are ingested, you can then define billable metrics over them, retaining the flexibility to evolve the way you charge without running a backfill.

Querying over rollup events

Since Orb’s rollups feature also outputs intermediate rollups, it’s important that these are accounted for in the billable metric queries that you form over them, effectively by executing a GROUP BY over the relevant properties and the grouping. Orb provides ergonomic syntax for querying over aggregated events as a “view”, abstracting this rollup mechanism away. For example, the following query:

SELECT SUM(aggregated_count)
FROM (
    SELECT MAX(count) as agg_count
    FROM events
    WHERE event_name = 'API_REQUEST_aggregated'
    AND data_status = 'success'
    GROUP BY
        timestamp_millis,
        data_region,
        data_clusterId
)

can be written instead as:

SELECT SUM(aggregated_count)
FROM AGGREGATED_EVENTS('API_REQUEST')
WHERE data_status = 'success'

Get started

Architecture

Essentials

Events and metrics

Subscription management

Usage alerting

Invoicing

Collections

Revenue reporting

Data exports

Simulations

Extending Orb

Hosted rollups

Architecture and configuration

Example: Rollups on data download

Querying over rollup events

​Architecture and configuration

​Example: Rollups on data download

​Querying over rollup events

Architecture and configuration

Example: Rollups on data download

Querying over rollup events