Data Orchestration
This article details the data orchestration process, how to modify the configuration of data orchestration, how to launch data orchestration manually, and how to interpret the details of a data orchestration.
What is data orchestration?
Data orchestration is a sequence of flows that performs data integration, lifecycle management, consolidation, computation, and assigned automations to get your data ready for searching and graphing. Data orchestration collects, organizes, and standardizes your data from all sources to provide a consolidated view of the data imported into the Brinqa Platform. While data integration maps the data retrieved from your connectors to data models, data orchestration generates metrics and calculates risk scores on your data to help you assess and prioritize your risks. Data orchestration and data integration work together to get your data ready for searching and visualization.
As shown in Figure 1, data integration creates the source data models (SDM) and then maps them to the unified data models (UDM). After consolidation, computation runs to generate risk scores and metrics on your assets and findings, as well as update your existing remediation requests, reports, and tickets. Brinqa Platform then takes a snapshot of the processed data and stores it in the Data Warehouse. Additionally, flexible workflows called automations are set up to run across different stages.
Figure 1. The data orchestration process.
The following table outlines the various stages in data orchestration:
| Stage | Description |
|---|---|
| Data Integration | Imports data from various sources through a streamlined and automated method. |
| Data Lifecycle | Enables you to designate data as inactive if it hasn't been updated within a set timeframe. Inactive data can then be scheduled for automatic purging, ensuring that only relevant and current data is retained in the Brinqa Platform. |
| Data Consolidation | Combines data from multiple sources into a unified format, eliminating duplicates and enriching the data. |
| Data Computation | Updates data by performing certain actions such as executing scripts, launching syncs, or sending notifications. |
| Data Storage | Processes Analytics Source and takes a snapshot of the unified data to stores in Data Warehouse. |
Change the data orchestration settings
To modify the schedule of data orchestration, select an action in the event of data integration failure, or notify users of successful or failed orchestrations, follow these steps:
-
Navigate to Administration
> Data > Orchestration. -
Click Edit and modify the following fields:
-
Schedule: By default, the data orchestration runs once per day. You can change how often the data orchestration runs, what time and day it runs, and the time zone.
-
Failure handling: Indicate what action to take upon a failed data integration or automation in the data orchestration process. By default, data orchestration continues to run and collect data from data integrations or automations when one of them fails. You can also stop the data orchestration altogether. Click the drop-down and select Stop orchestration to do so.
-
(Optional) Enable notifications. If needed, select Enable notifications and specify the roles or users to notify when the data orchestration runs successfully, fails, or runs successfully when a data integration or automation was skipped due to a failure.
-
-
Click Update.
Launch a data orchestration
Data orchestration runs daily at the scheduled time of your choice. To manually launch data orchestration, follow these steps:
-
Navigate to Administration > Data > Orchestration.
-
Click Launch.
Data orchestration may take a considerable amount of time and system resources to process depending on how many integrations you have configured and the scope of those integrations. If a data orchestration is in progress, you must cancel it before launching a new run.
Data orchestration details and logs
To see the logs or details of each data orchestration run, navigate to Administration > Data > Orchestration, hold your pointer over the data orchestration entry and click View logs or Details.
The Details page contains information on the steps and flows involved in the data orchestration.
Click the caret menu 
to expand the entry and view details. If a flow encounters an error and fails, you can see which step has failed.
The View logs option directs you to the Logs page containing logs generated for that specific data orchestration run. See Application Event Logs for additional information.
Brinqa removes both successful and failed flows after 28 days.
The Refresh button 
is disabled until the Data Orchestration details page fully loads.