Parquet Connector
File- Overview
- Setup
- Data & mappings
- Operations & API
- Changelog
The Parquet connector reads Apache Parquet files from a directory the Brinqa agent can reach and emits one record per row, mapping the file's columns onto a data model you choose. It completes the set of schema-agnostic file connectors alongside the CSV and JSON connectors.
Parquet is the default export format for Spark, Databricks, Athena, Snowflake unloads and pandas, so it is usually what security and asset inventory data already looks like in a customer data lake.
The connector derives its schema, and every column's data type, from the Parquet file itself. A Parquet file carries a typed schema in its footer, so unlike the CSV connector there is nothing to declare by hand: no delimiters, no text qualifier, no character encoding, and no lists of which columns are numbers, dates or booleans. Timestamps arrive as timestamps and decimals as decimals.
| Capability | Supported |
|---|---|
| Compression | Snappy, GZIP, ZSTD, LZ4 (raw and legacy), uncompressed |
| Nested data | Structs, lists, maps — see Nested columns |
| Column selection | Yes, at top-level column granularity |
| Multiple files | Yes, including files whose schemas differ |
| Incremental sync | Yes, by file modification time |
Data retrieved from parquet
| Connector Object | Required | Maps to Data Model |
|---|---|---|
| Record (configurable target) | Yes | Record (configurable target) |
Model relationships
For detailed steps on how to view the data retrieved from parquet in the Brinqa Platform, see How to view your data.
The agent needs write access to the data directory, not just read. Files are renamed as they are processed, and that is what stops the same data being ingested twice.
INT96 timestamps carry no time zone. Spark writes them in UTC, but Impala and older versions
of Hive write local wall-clock time, and nothing in the file distinguishes the two. If your
timestamps are off by a fixed number of hours, set Legacy timestamp time zone to the zone the
file was written in.
A list of structs becomes JSON rather than being flattened because flattening cannot record which
values belonged to the same element. Splitting orders into orders.sku = [A, B] and orders.qty
= [1, 2] loses the pairing, and there is no way to recover it. Keeping each element as one JSON
object keeps sku and qty together. Flatten lists of structs overrides this if each field is
meaningful on its own.
Connection settings
When setting up a data integration, select parquet from the Connector dropdown and provide the following:
| Setting | Required | Default | Description |
|---|---|---|---|
| Data directory (dataDirectory) | Yes | — | Path to the Parquet files. Wildcards allowed, e.g. /feed/exports/assets_*.parquet. Several comma separated patterns may be given. |
| Target (objectClassType) | Yes | Record | The data model name the rows map to. |
| Identifier fields (identifierFields) | No | — | Columns used to match records against existing data. Declared order matters. |
| Unique fields (uniqueFields) | No | — | Columns that together identify a row. When empty, the identifier is a hash of every value in the row. |
| Multi-row fields (multiRowFields) | No | — | Columns whose values are spread across several rows for the same record. Requires Unique fields. |
| Include columns (includeColumns) | No | — | Only these columns become attributes. Unselected top-level columns are never decompressed. |
| Exclude columns (excludeColumns) | No | — | Columns to ignore, applied after Include columns. |
| Nested column handling (nestedMode) | No | FLATTEN | FLATTEN, JSON or SKIP. |
| Maximum nesting depth (maxNestingDepth) | No | 5 | Levels of nested structs to flatten; deeper ones become JSON. |
| Maximum attributes (maxFlattenedAttributes) | No | 512 | Upper bound on discovered attributes. |
| Legacy timestamp time zone (int96TimeZone) | No | UTC | Time zone for INT96 columns. |
| Schema sample size (schemaSampleFileCount) | No | 10 | Files inspected to build the attribute list; 0 means all. |
| Strict schema (strictSchema) | No | false | Stop the sync on a type conflict instead of widening. |
| Max age (maxAgeInDays) | No | -1 | Days a processed file is retained. Below 0 keeps forever. |
| Max files (maxNumberOfFiles) | No | -1 | Processed files to retain. Below 0 keeps all. |
| Rename processed files (renameProcessed) | No | true | Rename each file after reading it. |
| Flatten lists of structs (flattenListOfStructColumns) | No | false | Flatten list-of-struct fields into parallel multi-valued attributes. Does not preserve element pairing. |
Authentication
The Parquet connector reads a filesystem rather than a vendor API, so it has no API credentials — no API key, token, OAuth flow, or authentication endpoint. What it authenticates against instead is the filesystem: the Brinqa agent process must be able to reach the configured Data directory.
| Element | Value |
|---|---|
| Method | None — local or mounted filesystem access. No API authentication. |
| Credential | None held by the connector. Access is granted by the operating system to the agent process. |
| Required access | Read and write on the Data directory and its files — write because each file is renamed (.locked, then .processed or .failed) as it is ingested. A read-only agent cannot sync. |
| Connection test | No auth handshake. The connector locates at least one readable Parquet file under the Data directory and reads its footer, failing with a clear error if none is found. |
Storage backends
The connector opens the Data directory as ordinary local paths, so data on a remote or cloud store must be mounted into the agent's filesystem first — the connector does not sign in to those services itself:
| Backend | How to expose it to the connector |
|---|---|
| Local disk / attached volume | Point Data directory at the path directly. |
| NFS / SMB network share | Mount on the agent host, then point Data directory at the mount. |
| Cloud object storage (Amazon S3, Azure Blob, Google Cloud Storage) | Surface it as a filesystem — a CSI volume, s3fs/blobfuse/gcsfuse, or a storage gateway — then point Data directory at that mount. |
How to obtain parquet credentials
Obtain the required credentials (Data directory (dataDirectory), Target (objectClassType)) from your parquet administrator or the parquet admin console, then enter them in the connection settings above.
Any credentials those mounts require — IAM roles, access keys, share passwords — are configured at the operating-system or infrastructure layer where the mount is created, outside the connector. The connector never sees, stores, or transmits them.
On wide files, Include columns is the single most effective setting. Parquet stores data by column, so an unselected top-level column is never read from disk at all.
Selection is applied per top-level column. Selecting address.city still reads the whole
address struct, because a struct is projected as a unit. Excluding address entirely does skip
it. This matters only for wide structs; top-level scalar columns are skipped precisely.
The connector buffers one row group at a time in memory, decompressed. Exports written by Spark default to 128 MB row groups, which can decompress to several hundred megabytes. Size the agent's heap accordingly, or use Include columns to narrow what is read. Each row group's row count is logged as it is read.
Attribute mappings
Expand the sections below to view the mappings between the source and the Brinqa data model attributes:
Record (configurable target)
| Source Field Name | SDM Attribute |
|---|---|
| Unique fields, or every value in the row | UID |
| :--------------------------------------------- | :---------------- |
| A field inside a struct column | address.city |
| Same as UID | NAME |
| Source Field | Attribute Name |
| The Parquet column | (column name) |
| The source file name, without its extension | FILE_NAME |
| The source file's modification time | MODIFIED |
| The time the sync started | LAST_CAPTURED |
A column named uid is reserved and is exposed as parquet_uid instead. The same applies to any
column colliding with one of the attributes above.
Operations & API
Expand each connector object to see its operation options, delta-sync behavior, and the API it uses. See connector operation options for how to apply operation options (keys and values are case-sensitive).
Record (configurable target)
Operation options
This object does not support any operation options.
Delta sync
Not supported. The connector performs a full sync of Record (configurable target) on every run and applies no incremental date filter.
API
- Type: See connector README
Changelog
The parquet connector has undergone the following changes:
| Version | Description | Migration Steps |
|---|---|---|
| 3.0.1 | No changes in this release. | N/A |
| 3.0.0 | Overview Initial release of the Parquet connector, which reads Apache Parquet files from a directory and loads one record per row into a data model of your choosing. Category: File Models New Features Schema and Types Read From the File Attribute names and data types are taken from the Parquet file's own schema. Nothing needs to be declared by hand — no delimiters, no character encoding, and no lists of which columns hold numbers, dates or booleans. Timestamps load as timestamps and decimals as decimals, at full precision. Compression Support Files compressed with Snappy, GZIP, ZSTD or LZ4 are read directly, as are uncompressed files. Snappy is what most tools write by default. Nested Columns Fields inside a struct become individually mappable attributes with dotted names, such as address.city. Lists of values become multi-valued attributes. Lists of structs and maps are provided as JSON so that the values belonging to each element stay together. Nested column handling switches all nested columns to JSON, or skips them entirely. Column Selection Include columns and Exclude columns narrow what is loaded. Because Parquet stores data by column, an unselected top-level column is never read from disk, which makes a noticeable difference on wide files. Files With Different Schemas A directory frequently holds exports written at different times, where a newer export has added a column. The connector combines the schemas of the files it finds, so a column present in only some files still appears, and a column whose type changed between files is widened to a type that holds both. Strict schema stops the sync on such a conflict instead, for cases where being told matters more than continuing. Multi-Row Records Multi-row fields combines rows that describe the same record, which is the shape of exports carrying one row per asset and finding pair. Legacy Timestamp Time Zones Older INT96 timestamp columns carry no time zone, and while Spark writes them in UTC, Impala and older versions of Hive write local time. Legacy timestamp time zone sets which to assume, since the file itself gives no indication. Resilience A file that cannot be read is set aside and logged without stopping the rest of the directory from loading. Empty files load successfully and contribute no records. Migration No migration steps required — this is the initial release. | N/A |