Skip to content

Commit

Permalink
Add comment.
Browse files Browse the repository at this point in the history
Signed-off-by: Revital Sur <[email protected]>
  • Loading branch information
revit13 committed Mar 31, 2022
1 parent 7db183e commit 226a8aa
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions afm/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@ def _write_asset(self, asset, reader, write_mode):
# in this implementation we currently begin by reading the entire dataset
record_batches = reader.read_all().combine_chunks().to_batches()
transformed_batches = transform_batches(asset.actions, record_batches)
# If the client request to append to the existing data then the flag `overwrite_or_ignore` is used
# with a basename_template that is related to the time of writing. This operation writes the data
# to a new file while ignoring (not changing) the existed files.
# Otherwise, the flag `delete_mathing` is used. This operation writes the data to a new file
# while deleting the existed files.
if write_mode == "append":
logger.trace("write_mode: append", extra={DataSetID: asset.name, ForUser: True})
ds.write_dataset(transformed_batches, base_dir=asset.path, basename_template="part-{:%Y-%m-%d-%H-%M-%S-%f}-{{i}}.parquet".format(datetime.datetime.now()), format=asset.format, filesystem=asset.filesystem, existing_data_behavior='overwrite_or_ignore')
Expand Down

0 comments on commit 226a8aa

Please sign in to comment.