Load Data from Parquet file to PostgreSQL

[MahmoudHousam]

Parquet files are read as batches, pulled to Pandas df and then to a PostgreSQL DB. This process takes up to 3 minutes to pull 3+ million rows from Parquet to PostgreSQL.

Recommended Fix: Use Duckdb and duckdb_fwd to read the Parquet file directly from the source and push it to PostgreSQL DB. (under investigation)

Alternative: Use Airbyte Cloud

Resources:

• Loading Parquet in PostgreSQL via DuckDB: Testing queries and exploring the Core

• How to load data from Parquet file to PostgreSQL destination