Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bqetl query backfill does not support non-partitioned tables #4532

Closed
fbertsch opened this issue Nov 7, 2023 · 1 comment · Fixed by #4769
Closed

bqetl query backfill does not support non-partitioned tables #4532

fbertsch opened this issue Nov 7, 2023 · 1 comment · Fixed by #4769
Assignees

Comments

@fbertsch
Copy link
Contributor

fbertsch commented Nov 7, 2023

The table fenix_derived.client_adclicks_history_v1 is not partitioned, but I would like to backfill it by-day:

bqetl query backfill fenix_derived.client_adclicks_history_v1 --sql_dir sql --project_id moz-fx-data-shared-prod --start_date 2020-08-01 --end_date 2023-11-07 --max_rows 4 --parallelism 1 --checks

However the above fails with:

  File "/Users/fbertsch/repos/bigquery-etl/bigquery_etl/cli/query.py", line 725, in backfill
    partitioning_type = metadata.bigquery.time_partitioning.type
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'type'

Looking through the code, we hardcode the partitioning spec in several places (it requires daily or monthly in both backfill and _backfill_query.

┆Issue is synchronized with this Jira Task

@ANich ANich self-assigned this Nov 29, 2023
@ANich
Copy link
Contributor

ANich commented Nov 29, 2023

I've been looking into this on and off as a part of the managed backfills work. There's also a couple other things that aren't by default supported:

  • date_partition_offset (technically possible iff there's also a date_partition_parameter)
  • Queries with additional parameters

I'm working on a few suggestions/PRs. I'll link this issue where relevant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants