Aqueducts v0.8.0

This release is upgrading to Datafusion v44 and deltalake 0.23 and includes TOML as a new serialization format.
Additionally all the serialization libraries are feature toggled and can be installed by setting the feature flag for the respective format on install.

0.8.0 - 2025-01-08

Details

Added

• Add support for toml format and make serialization libraries toggelable through features by @vigimite

Changed

• Update datafusion to v44 and delta-rs to 0.23 by @vigimite
• Change delta table merge to use schema mode overwrite instead of merge by @vigimite

Fixed

• Failing test cases with disabled json feature by @vigimite
• Enable yaml as default serialization for aqueducts-cli by @vigimite

Aqueducts v0.7.1

This release only contains dependency updates, a major one being a delta-rs fix that fixes a breaking bug in that crate.

0.7.1 - 2024-12-09

Details

Changed

• Bump deltalake from 0.22.0 to 0.22.3 by @dependabot[bot] in #51

Aqueducts v0.7.0

0.7.0 - 2024-11-29

Details

Changed

• Upgrade datafusion to v43 and deltalake to v0.22 by @vigimite
• Downgrade serde_yml version to fix broken libyml implementation by @vigimite
• Bump thiserror from 1.0.69 to 2.0.3 by @dependabot[bot] in #48

Aqueducts v0.6.3

This release just updates a bunch of dependencies

0.6.3 - 2024-10-31

Details

Changed

• Bump thiserror from 1.0.64 to 1.0.65 by @dependabot[bot] in #35
• Bump serde from 1.0.210 to 1.0.213 by @dependabot[bot] in #34
• Bump tokio from 1.40.0 to 1.41.0 by @dependabot[bot] in #32
• Bump serde_json from 1.0.128 to 1.0.132 by @dependabot[bot] in #31
• Bump clap from 4.5.18 to 4.5.20 by @dependabot[bot] in #30
• Update deltalake dependency @vigimite

Fixed

• Failing test by @vigimite

Aqueducts v0.6.2

Highlights

This is a patch release to fix the case of an ODBC transaction failing not reporting the error upstream

0.6.2 - 2024-09-30

Details

Changed

• Downgrade arrow_odbc version by @vigimite
• Downgrade arrow_odbc version (12.0.0) by @vigimite

Fixed

• Fix lockfile by @vigimite
• Failed transaction returns Ok

Aqueducts v0.6.1

Highlights

This is a patch release to fix handling of arrow primitives for the ODBC insert cases

0.6.1 - 2024-09-27

Details

Changed

• Bump clap from 4.5.17 to 4.5.18

Bumps clap from 4.5.17 to 4.5.18.

• Release notes
• Changelog
• Commits

---

updated-dependencies:

• dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
  ...

Signed-off-by: dependabot[bot] [email protected]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> by @dependabot[bot] in #27

• Bump thiserror from 1.0.63 to 1.0.64

Bumps thiserror from 1.0.63 to 1.0.64.

• Release notes
• Commits

---

updated-dependencies:

• dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
  ...

Signed-off-by: dependabot[bot] [email protected]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> by @dependabot[bot] in #26

• Bump arrow-odbc from 12.1.0 to 12.2.0

Bumps arrow-odbc from 12.1.0 to 12.2.0.

• Changelog
• Commits

---

updated-dependencies:

• dependency-name: arrow-odbc
  dependency-type: direct:production
  update-type: version-update:semver-minor
  ...

Signed-off-by: dependabot[bot] [email protected]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> by @dependabot[bot] in #25

• Merge branch 'main' of https://github.com/vigimite/aqueducts

Fixed

• Incorrect record batch handling for custom ODBC insert case by @vigimite
• Incorrect record batch handling for custom ODBC insert case (2nd case)

Aqueducts v0.6.0

Highlights

This release introduces a custom write mode for ODBC destinations. This allows for more complex data insertion possibilities like doing an upsert or overwriting partitions of data.

Example of custom write mode:

sources:
  - type: InMemory
    name: existing_in_context

stages:

...

  - - name: result
      query: SELECT * FROM aggregated

destination:
  type: Odbc
  name: temps
  write_mode:
    operation: Custom
    # all of the statements are executed in a single transaction including the insert, on failure a rollback is triggered
    transaction:
      # one or more statements to execute before inserting all of the records
      pre_insert: >
        delete from temps where location_id = 1;
        delete from temps where location_id = 2;
      
      # statement used for insertion of records
      # the parameters are going to be applied sequentially using the schema of the `result` dataframe to determine each col position
      # this also supports indexed positional arguments if the ODBC driver supports it (e.g. `$1, $2, $3`)
      insert: >
        insert into temps values
          (?, ?, ?, ?, ?, ?);

  batch_size: 50
  connection_string: Driver=PostgreSQL;Server=localhost;Port=5432;Database=postgres;UID=postgres;PWD=postgres;

0.6.0 - 2024-09-17

Details

Changed

• Updated deps, make destination name public, include ')' for ttl check by @vigimite
• Add Custom write mode for ODBC destination by @FGahan in #23

Removed

• Remove object_store as a dependency (unused) by @vigimite

Aqueducts v0.5.0

Highlights

Support for In Memory source and destination. The in Memory source was already implicitly possible by passing in a context however now there is the added possibility to reference it directly in the aqueducts definition which checks for the existence of the source before running the pipeline. In addition to this the run_pipeline function now returns the provided context again which allows further re-use of the context and consume any table that was output into memory downstream.

Example in memory source:

sources:
  - type: InMemory
    name: existing_in_context

stages:

...

  # last stage will be kept in the context
  - - name: result
      query: SELECT * FROM aggregated

stages
destination:
  type: InMemory
  name: temp_readings_aggregated

0.5.0 - 2024-08-20

Details

Added

• Add InMemory source and destination by @vigimite

Changed

• Update release workflow by @vigimite
• Update schema docs to include ODBC destinations by @vigimite
• Bump datafusion to v40 by @vigimite
• Bump datafusion to v41 and delta-rs to 0.19.0 by @vigimite
• Update object_store requirement from 0.10 to 0.11

Updates the requirements on object_store to permit the latest version.

• Changelog
• Commits

---

updated-dependencies:

• dependency-name: object_store
  dependency-type: direct:production
  ...

Signed-off-by: dependabot[bot] [email protected]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> by @dependabot[bot] in #20

• V0.5.0 by @vigimite

Aqueducts v0.4.0

Highlights

Newly introduced ODBC support for destinations. This destination type will try to insert records into an existing DB table using arrow_odbc to convert the arrow types into the corresponding column types.
Support for upserting or replacing records in a table will come in the next minor releases.

Example ODBC destination:

sources:

...

stages:

...

  # last stage will be inserted into the target table
  - - name: result
      query: SELECT * FROM aggregated

stages
destination:
  type: Odbc
  name: temp_readings_aggregated
  connection_string: Driver={PostgreSQL Unicode};Server=localhost;UID=${user};PWD=${pass};
  batch_size: 1000

0.4.0 - 2024-07-28

Details

Added

• Add support for ODBC destinations by @vigimite in #17

Changed

• Make json schema generation optional by @vigimite

Aqueducts v0.3.2

0.3.2 - 2024-07-06

Details

Changed

• Revert "prepare release v0.3.1"

This reverts commit 38d4980. by @vigimite

• Prepare release v0.3.2 by @vigimite

Fixed

• Don't coerce LargeUtf8 and LargeBinary into smaller types by @vigimite

Removed

• Revert "fix: remove schema validation due to incorrect casting of LargeUtf8"

This reverts commit a924aaf. by @vigimite