A value of 'big_query_array' is being inserted into BigQuery schemas as part of BigQuery Denormalized Destination 0.2.0 and above.

[zbrak]

Environment

• Airbyte version: example is 0.32.8-alpha
• OS Version / Instance: Google Cloud 1.21.6-gke.1500
• Deployment: GKE
• Source Connector and version: NA ; Asana 0.1.3 used for example.
• Destination Connector and version: bigquery-denormalized 0.2.10
• Severity: Critical
• Step where error happened: Update Connector

Current Behavior

Values of big_query_array are being added into table schemas.

Expected Behavior

Erroneous values should not be entered into table schemas.

A large part of what makes the bigquery-denormalized destination valuable is that it holds schema fidelity between the original JSON source and the BigQuery destination table.

By interjecting these big_query_array values throughout the schema, the schema fidelity is degraded. There are downstream problems as well, as this effectively corrupts the dot-notation paths that we need to reference in both macros and documentation.

This connector is extremely special and useful, I think it is undervalued given the presence of the standard BigQuery connector.
(i.e. this bigquery-denormalized should be the 'standard connector')

Note that a denormalized strategy is more performant in BigQuery.

Logs

Steps to Reproduce

1. Sync Asana 'Projects' stream
2. View output schema in BigQuery

Are you willing to submit a PR?

Do not have dev resources to put to this yet - should have dev resources available in late May.