Data Integration Overview

This article details the process of getting your data into the Brinqa Platform through connectors.

What is data integration?

Data integration in a Brinqa Platform entails the process of importing your data through a streamlined and automated method. Data integration involves configuring a connector and indicating how far back in time you want to retrieve your data.

The Integrations tab brings you to the data integration page and the following table describes the columns on the page:

Table 1: Data integration list view columns

Column	Description
State	Whether the data integration is connected or disconnected.
Title	The name of your data integration.
Connector	The connector used for your data integration.
Version	The version the connector is on. To update an installed connector, navigate to Integrations > Connectors, search for that specific connector, and click Update.
Server	The data server on which the data integration is running.
Last sync start	When the data integration was last synced.
Duration	Shows how long it takes for the data integration sync to run and complete.
Updated by	By whom the data integration was last updated (individual or the system)
Last updated	When the data integration configuration was last updated.

Click the title of the data integration to see more details about the data the connector collects. The data integration details page contains different columns with information pertaining to your data integration, as shown in the following table:

Table 2: Data integration show page details

Column	Description
Status	Indicates where the data integration sync is at in terms of completion. Examples include successful , in progress , or failed .
Transaction ID	Displays a 36-character string with a mix of numbers and letters that represents the data integration. You can copy and paste the Transaction ID into the Application Event Log search bar to troubleshoot and view details.
Source	Lists each data model configured for the data integration.
Started	Shows the time the data integration sync starts.
Creates	Shows the number of such items the data integration sync creates. For example, a connector that retrieved 8.4 million vulnerabilities after a sync.
Updates	Shows the number of changes in the data compared to the last data integration sync.
Duration	Shows how long it takes for the data integration sync to run and complete.

Create a data integration

If you have already configured a data integration and are comfortable with the steps in creating data integrations, you can skip to Run the data integration.

To create a data integration, follow these steps:

1. Navigate to Integrations and click Create.

   • You can also navigate to Integrations > Connectors and click Use on the connector after it is installed. The Connector field in the data integration information section is filled out with the connector you have selected.

2. Fill out the general information:

   • Title: Title of your data integration.

   • Connector: The connector to use in the data integration.

   • Server: The server to process data in the data integration. Local server is selected by default for cloud data sources. You can also create your own data servers.

   • Description: Description of the data integration, for example, a list of the services or data it provides.

3. Complete the configuration for the connector. Specific fields may differ based on the connector:

   • Server URL: The Uniform Resource Locator (URL) to access the data source.

   • Username: The username for the specific connector account. Brinqa does not provide vendor login credentials.

   • Password: The password for the specific connector account. Brinqa does not provide vendor login credentials.

   • API Keys/Tokens: Codes used to authenticate the connector. Certain connectors have configuration settings that replace the Username and Password fields with API keys. For example, the Tenable.io connector requires an Access Key and a Secret Key.

     info

     If you are unsure of the correct server URL, credentials, or API keys for the connector, refer to the specific connector documentation or contact an administrator.

4. (Optional) Select Advanced. Some connectors contain advanced options for specific configuration, for example, to set the page size (maximum number of records to get per API request) or parallel requests (maximum number of parallel API requests).

5. Click Next to save the configuration. If the connection settings are correct, more options display.

6. In Types, you can view the types of data the connector retrieves. For example, the Qualys Vulnerability Management connector brings in data for Host, Vulnerability, and Vulnerability Definition.

   • Click the entry to see the attributes associated with the data model. For example, the screenshot below shows some of the attributes the Vulnerability data model provides:

   

   caution

   While selecting additional attributes for your connector may be beneficial, the more attributes you select the longer it takes for that specific data integration to sync and complete.

7. (Optional) Configure operation options for the data model. See connector operation options for additional information.

8. Set the sync interval. Sync interval determines how far back you want to sync your data. The default setting is the beginning of time, which refers to the Unix epoch of January 1st, 1970 at 00:00:00 UTC. This essentially means when your Brinqa Platform is originally deployed.

   • Click the entry to select from a range of options.

     tip

     Brinqa recommends that you use the beginning of time option when you run the data integration initially to import all your data thus far. After the first sync, change the sync interval to the last sync to save time and resources.

9. Click Create.

If the connection is successful, the page reloads and you should see your new data integration listed. If you do not see it, click Refresh .

If the connection is not successful, the data integration is not created. Double-check the authentication credentials to ensure they are correct.

Connector operation options

Some connectors support operation options, which are used by the connector to build filters when retrieving data. Operation options can impact the amount of time for data integration to complete and help find more targeted data. Operation options are connector-specific and can only be applied if the vendor API supports them. See individual connector documentation for more information.

You can define operation options for the type of data you want the connector to retrieve. For example, in step 6 of the previous instructions, follow these steps to limit the Vulnerability data to active vulnerabilities:

• In Types, without expanding the attributes, hold the pointer over Vulnerability and click Options. A new window appears.

  • For Key, type Status; for Value, type Active. Operation options are case-sensitive.

  • Click Save.

Take some time to get familiar with the different data models, attributes, and connector option keys for your connector and decide which ones benefit you in your data integration. You may want to add more attributes or set certain operation options to ensure your system is populated with the data you want to bring in.

Run the data integration

Now that you have created your data integration, you can start running syncs to gather data. By default, the data integration runs once a day automatically through data orchestration. You can sync your data manually as well.

Hold the pointer over the data integration until you see a few options appear on the right-hand side. These options are Sync, Show, Edit, and a kebab (three vertical dots) menu. Click Sync to manually sync and run your data integration. There are options to select how far back you want to sync your data. See Step 8 in Create a data integration for more information.

You can also perform a manual sync of your data by clicking Sync on the data integration details page. Both methods run the data integration sync to bring your data into the Brinqa Platform. The time it takes for the sync to complete depends on a multitude of factors. For example, the number of data models receiving data, the number of attributes, the number of filters, or the amount of data coming in. Therefore, the sync can take a few minutes or up to a few hours.

If you have multiple data integrations, it is recommended that you do not run syncs concurrently. Allow one sync to finish before performing further syncs.

Edit or delete a data integration

Navigate to Integrations. Hold the pointer over the data integration until you see a few options appear on the right-hand side. These options are Sync, Show, Edit, and a kebab (three vertical dots) menu. The kebab menu contains options to configure the mapping (if applicable), test the connection, delete the data integration, or cancel the data integration sync if it is running.