Data exports
Orb supports managed, daily exports of resources to your data destination of your choice. Typically, this is a data warehouse such as Redshift or Snowflake, but other types of data stores are also supported. Exporting data from Orb can be done directly within the Orb UI; you can configure data exports directly within the UI.
Data exports are not included with all Orb billing plans by default. Please reach out to your Orb representative if you have any questions.
Export mechanics
Orb's data exports are managed by Orb. This managed solution exports data directly to your destination of choice.
Supported destinations
Orb supports the following data destinations:
- BigQuery
- Redshift
- Amazon S3
- Snowflake
We're actively working to support more destinations. If you don't see your destination of choice listed, please reach out to your Orb representative.
How Orb syncs changes
For nearly all resources, Orb provides an incremental sync, which significantly reduces the data that is synced on a regular basis. The first sync that is run on the data exporter is a full sync (and may transfer a large number of resources). Future syncs after this are incremental and will only capture the updates that occur.
In Orb's system, changes to the core resources (e.g. creations, updates, or deletes) are tracked internally. These records are used to help understand what changed and when. Using these records, Orb's system is able to power the logic that determines which resources need to be exported in a given timeframe during an incremental sync.
Orb's internal process has two, dependent stages:
- Serializing / dumping resources in preparation for the transfer process
- The transfer process where these resources are queried then copied to the end data store
Export UI
Note that only account admins are able to access the data exports UI and interact with data export connections.
The UI for the Data Export product can be found under "Data exports" within the Settings page.
A new data export connection can be added by clicking the "Create connection" button and filling out information about your data destination.
The information needed about your data destination will different depending on which destination system you're choosing. The UI within Orb will dynamically provide you with the relevant inputs needed to configure a data export.
After filling in the required information about your data destination, you will be able to choose which data models you want to sync.
After creating a new data export connection, you'll be able to view it directly in the UI.
The UI also provides the ability to enable, disable, or delete your data export destination depending on its current state. Use the three vertical dots to see these options. The current status of the destination is also listed within the table and will appear differently if it is disabled.
