Ingest events

Orb’s event ingestion model and API is designed around two core principles:

1. Data fidelity: The accuracy of your billing model depends on a robust foundation of events. Orb’s API protocol encourages usage patterns that ensure that your data is consistently complete and correct.
2. Fast integration: Sending events into Orb requires no tedious setup steps or explicit field schema for your event shape, making it instant to start streaming in usage in real-time.

Event shape

Events are the starting point for all usage calculations in the system, and are simple at their core:

{
  // customer_id and external_customer_id are used to
  // attribute usage to a given Customer. Exactly one of these
  // should be specified in a given ingestion event.

  // `customer_id` is the Orb generated identifier for the Customer,
  // which is returned from the Create customer API call.
  customer_id: string,

  // external_customer_id is an alternate identifier which is associated
  // with a Customer at creation time. This is treated as an alias for
  // customer_id, and is usually set to an identifier native to your system.
  external_customer_id: string,

  // A string name identifying the event, usually a usage
  // action. By convention, this should not contain any whitespace.
  event_name: string,

  // An ISO 8601 format date with no timezone offset.
  // This should represent the time that usage occurred
  // and is important to attribute usage to a given
  // billing period. See the notes below on determining the timestamp.
  // e.g. 2020-12-09T16:09:53Z
  timestamp: string,

  // A unique value, generated by the client, that is
  // used to de-duplicate events.
  // Exactly one event with a given
  // idempotency key will be ingested, which allows for
  // safe request retries.
  idempotency_key: string

  // Optional custom metadata to attach to the event.
  // This might include a numeric value used for aggregation,
  // or a string/boolean value used for filtering.
  // The schema of this dictionary need not be pre-declared, and
  // properties can be added at any time.
  properties: {
    [key: string]?: string | number | boolean,
  },
}

Required fields

Because events streamed to Orb are meant to be as flexible as possible, there are only a few required fields in every event.

• We recommend that idempotency_key are unique strings that you generated with V4 UUIDs, but only require that they uniquely identify an event (i.e. don’t collide).
• The timestamp field in the event body will be used to determine which billable period a given event falls into. For example, with a monthly billing cycle starting from the first of December, Orb will calculate metrics based on events that fall into the range 12-01 00:00:00 <= timestamp < 01-01 00:00:00.

Logging metadata

Orb allows tagging events with metadata using a flexible properties dictionary. Since Orb does not enforce a rigid schema for this field-set, key-value pairs can be added dynamically as your events evolve.

This dictionary can be helpful for a wide variety of use cases:

• Numeric properties on events like compute_time_ms can later be inputs to our flexible query engine to determine usage.
• Logging a region or cluster with each event can help you provide customers more granular visibility into their usage.
• If you are using matrix pricing and matching a matrix price key with a property, you should ensure the value for that property is sent as a string.

We encourage logging this metadata with an eye towards future use cases to ensure full coverage for historical data. The datatype of the value in the properties dictionary is important for metric creation from an event source. Values that you wish to numerically aggregate should be of numeric type in the event.

Determining event timestamp

For cases where usage is being reported in real time as it is occurring, timestamp should correspond to the time that usage occurred.

In cases where usage is reported in aggregate for a historical timeframe at a regular interval, we recommend setting the event timestamp to the midpoint of the interval. As an example, if you have an hourly reporter that sends data once an hour for the previous hour of usage, setting the timestamp to the half-hour mark will ensure that the usage is counted within the correct period.

Note that other time-related fields (e.g. time elapsed) can be added to the properties dictionary as necessary.

In cases where usage is reported in aggregate for a historical timeframe, the timestamp must be within the grace period set for your account. Events with timestamp < current_time - grace_period will not be accepted as a valid event, and will throw validation errors. Enforcing the grace period enables Orb to accurately map usage to the correct billing cycle and ensure that all usage is billed for in the corresponding billing period.

In general, Orb does not expect events with future dated timestamps. In cases where the timestamp is at least 24 hours ahead of the current time, the event will not be accepted as a valid event, and will throw validation errors.

Event validation

Orb’s validation ensures that you recognize errors in your events as quickly as possible, and the API provides informative error messages to help you fix problems quickly.

We validate the following:

• Exactly one of customer_id and external_customer_id should be specified.
• If the customer_id is specified, the customer in Orb must exist.
• If the external_customer_id is specified, the customer in Orb does not need to exist. Events will be attributed to any future customers with the external_customer_id on subscription creation.
• timestamp must conform to ISO 8601 and represent a timestamp at most 1 hour in the future. This timestamp should be sent in UTC timezone (no timezone offset).

Idempotency and retry semantics

Orb’s idempotency guarantees allow you to implement safe retry logic in the event of network or machine failures, ensuring data fidelity. Each event in the request payload is associated with an idempotency key, and Orb guarantees that a single idempotency key will be successfully ingested at most once. Note that when Orb encounters events with duplicate idempotency keys and differing event bodies in a batch of events, the entire batch will be rejected.

• Successful responses return a 200 HTTP status code. The response contains information about previously processed events.
• Requests that return a 4xx HTTP status code indicate a payload error and contain at least one event with a validation failure. An event with a validation failure can be re-sent to the ingestion endpoint (after the payload is fixed) with the original idempotency key since that key is not marked as processed.
• Requests that return a 5xx HTTP status code indicate a server-side failure. These requests should be retried in their entirety.

API usage and limits

The ingestion API is designed made for real-time streaming ingestion and architected for high throughput. Even if events are later deemed unnecessary or filtered out, we encourage you to log them to Orb if they may be relevant to billing calculations in the future.

To take advantage of the real-time features of the Orb platform and avoid any chance of dropped events by producers, we recommend reporting events to Orb frequently. Optionally, events can also be briefly aggregated at the source, as this API accepts an array of event bodies.

Orb does not currently enforce a hard rate-limit for API usage or a maximum request payload size, but please give us a heads up if you’re changing either of these factors by an order of magnitude from initial setup.

Testing in debug mode

The ingestion API supports a debug mode, which returns additional verbose output to indicate which event idempotency keys were newly ingested or duplicates from previous requests. To enable this mode, mark debug=true as a query parameter.

If debug=true is not specified, the response will only contain validation_failed. Orb will still honor the idempotency guarantees set here in all cases.

We strongly recommend that you only use debug mode as part of testing your initial Orb integration. Once you’re ready to switch to production, disable debug mode to take advantage of improved performance and maximal throughput.

Example: ingestion response with debug=true

{
  "debug": {
    "duplicate": [],
    "ingested": [
      "B7E83HDMfJPAunXW",
      "SJs5DQJ3TnwSqEZE",
      "8SivfDsNKwCeAXim"
    ]
  },
  "validation_failed": []
}

Example: ingestion response with debug=false

{
  "validation_failed": []
}