Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatible table partitioning specification when copying to the column partitioned table #100

Closed
sonots opened this issue Jul 30, 2019 · 1 comment · Fixed by #101
Closed

Comments

@sonots
Copy link
Member

sonots commented Jul 30, 2019

If users try to replace a non-partitioned table table_name to a partitioned table as:

out:
  type: bigquery
  mode: replace
  table: table_name
  time_partitioning:
    type: DAY
    field: timestamp

it raises an error: Incompatible table partitioning specification when copying to the column partitioned table https://cloud.google.com/bigquery/docs/managing-partitioned-table-data

delete_in_advance also does not work now:

out:
  type: bigquery
  mode: delete_in_advance
  table: table_name
  time_partitioning:
    type: DAY
    field: timestamp

This is because of

if task['time_partitioning']
bigquery.delete_partition(task['table'])
It tries to delete a partition rather than a table.

@sonots
Copy link
Member Author

sonots commented Aug 10, 2019

Fixed via #101 and released v0.4.14. cc: @shierote

NOTE: Replacing a non-partitioned table table_name to a partitioned table as:

out:
  type: bigquery
  mode: replace
  table: table_name
  time_partitioning:
    type: DAY
    field: timestamp

is still not supported because BigQuery does not allow replacing atomically them.
Incompatible table partitioning specification when copying to the column partitioned table will be thrown. But, mode: delete_in_advance should work now as:

out:
  type: bigquery
  mode: delete_in_advance
  table: table_name
  time_partitioning:
    type: DAY
    field: timestamp

This is noted in README.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

1 participant