Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: handle case when expirationMs is None #1551

Conversation

abdelmegahedgoogle
Copy link

Right now, unsetting ['timePartitioning']['expirationMs'] on BigQuery table is not supported and raises google.api_core.exceptions.BadRequest: 400 PATCH https://bigquery.googleapis.com/bigquery/v2/projects/abdelrahmanm-bd-dev/datasets/partition_expiry_test/tables/sample_data_partitioned?prettyPrint=false: Invalid value at 'table.time_partitioning.expiration_ms.value' (TYPE_INT64), "None"

This happens since the backend doesn't treat "None" (string) as a null. This PR addresses this issue from the client side.

@google-cla
Copy link

google-cla bot commented Apr 14, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Apr 14, 2023
@abdelmegahedgoogle
Copy link
Author

Sample code that fails with the above exception:

from google.cloud import bigquery
client = bigquery.Client(project='abdelrahmanm-bd-dev')
table_ref = client.dataset('partition_expiry_test').table('sample_data_partitioned')
table = client.get_table(table_ref)
table.partition_expiration = None # ISSUE HERE!! This get cast to string and the backend returns a 400 as a result.
client.update_table(table, ['partition_expiration'])

@abdelmegahedgoogle abdelmegahedgoogle force-pushed the supportPartitionExpirationNoneValue branch from 78ce26a to 1497090 Compare April 17, 2023 19:35
plamut and others added 22 commits April 17, 2023 15:51
* feat: allow cell magic body to be a $variable

* Fix missing indefinitive article in error msg

* Adjust test assertion to error message change

* Refactor logic for extracting query variable

* Explicitly warn about missing query variable name

* Thest the query "variable" is not identifier case
…eapis#1073)

* feat: promote `to_arrow_iterable` to public method

* use correct version number

* Update google/cloud/bigquery/table.py

Co-authored-by: Tim Swast <[email protected]>
* fix: apply timeout to all resumable upload requests

* Fix stub in test case

* Improve timeout type and other type annotations

* Annnotate return type of _do_resumable_upload()
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
…taFrame (googleapis#1078)

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Towards #1076 🦕
(edit: moved to googleapis/python-db-dtypes-pandas#45 )
* revoke dataset access setup

* basic template for sample

* sample + test

* revoke dataset access sample

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* docs: add sample for revoking dataset access - update year and string formatting

* docs: add sample for revoking dataset access - move to snippets and change parameter pattern for readibility

* moving update_dataset to /snippets and adjusting imports on both revoke_access and update_access

* Update samples/snippets/revoke_dataset_access.py

removed nested START/END tags

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/revoke_dataset_access.py

update readability in API request

Co-authored-by: Tim Swast <[email protected]>

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* updated test

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/master/packages/owl-bot/README.md

* change after running test

* resolving linting failure, rewriting test

* removed relative import errors

* remove relative mport from update_dataset_access

* adding fixture to conftest.py

* updated sample

* updating sample to match new update_access sample

* fixing region tags

* consolidated tests into one file for both methods

* updating test to full_dataset format

* updated revoke sample

* updating test

* refactored sample

* Update samples/snippets/conftest.py

* Update samples/snippets/revoke_dataset_access.py

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/update_dataset_access.py

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/revoke_dataset_access.py

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/revoke_dataset_access.py

Co-authored-by: Tim Swast <[email protected]>

* refactoring entry

* added comment for entry access

* Update samples/snippets/README.rst

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/dataset_access_test.py

Co-authored-by: Tim Swast <[email protected]>

* Update samples/snippets/dataset_access_test.py

Co-authored-by: Tim Swast <[email protected]>

* added develper TODO in sample

* add comments to samples

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Tim Swast <[email protected]>
Co-authored-by: Peter Lamut <[email protected]>
Co-authored-by: Anthonios Partheniou <[email protected]>
Co-authored-by: meredithslota <[email protected]>
* feat: support authorized dataset entity

* cleanup

* add test and cache the resource from from_api_repr in a _properties value

* lint

* update samples to use enums

* update to_api_repr and add tests

* refactor
* chore(deps): update all dependencies

* exclude new geopandas from python 3.6

Co-authored-by: Tim Swast <[email protected]>
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [google-api-core](https://togithub.com/googleapis/python-api-core) | `==2.3.0` -> `==2.3.2` | [![age](https://badges.renovateapi.com/packages/pypi/google-api-core/2.3.2/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/google-api-core/2.3.2/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/google-api-core/2.3.2/compatibility-slim/2.3.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/google-api-core/2.3.2/confidence-slim/2.3.0)](https://docs.renovatebot.com/merge-confidence/) |
| [importlib-metadata](https://togithub.com/python/importlib_metadata) | `==4.8.2` -> `==4.8.3` | [![age](https://badges.renovateapi.com/packages/pypi/importlib-metadata/4.8.3/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/importlib-metadata/4.8.3/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/importlib-metadata/4.8.3/compatibility-slim/4.8.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/importlib-metadata/4.8.3/confidence-slim/4.8.2)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/python-api-core</summary>

### [`v2.3.2`](https://togithub.com/googleapis/python-api-core/releases/v2.3.2)

[Compare Source](https://togithub.com/googleapis/python-api-core/compare/v2.3.1...v2.3.2)

##### Bug Fixes

-   fix: exclude function target from retry deadline exceeded exception message ([#&#8203;318](https://www.togithub.com/googleapis/python-api-core/issues/318)) ([34ebdcc](https://togithub.com/googleapis/python-api-core/commit/34ebdcc251d4f3d7d496e8e0b78847645a06650b))

### [`v2.3.1`](https://togithub.com/googleapis/python-api-core/compare/v2.3.0...v2.3.1)

[Compare Source](https://togithub.com/googleapis/python-api-core/compare/v2.3.0...v2.3.1)

</details>

<details>
<summary>python/importlib_metadata</summary>

### [`v4.8.3`](https://togithub.com/python/importlib_metadata/blob/HEAD/CHANGES.rst#v483)

[Compare Source](https://togithub.com/python/importlib_metadata/compare/v4.8.2...v4.8.3)

\======

-   [#&#8203;357](https://togithub.com/python/importlib_metadata/issues/357): Fixed requirement generation from egg-info when a
    URL requirement is given.

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-bigquery).
* chore(deps): update all dependencies

* don't directly depend on cachetools

* remove importlib metadata as direct dependency

* remove zipp which comes from transitive dependency

Co-authored-by: Tim Swast <[email protected]>
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

To match the Trove classifiers on PyPI:

![image](https://user-images.githubusercontent.com/1324225/148646991-cc536f1f-4389-4cd9-9995-276c9aa7245a.png)

https://pypi.org/project/google-cloud-bigquery/
Source-Link: googleapis/synthtool@52aef91
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:36a95b8f494e4674dc9eee9af98961293b51b86b3649942aac800ae6c1f796d4

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…essage` instead (googleapis#1105)

Since query text can potentially contain sensitive information, remove it from
the default exception message. This information is useful for debugging, so
provide a `debug_message` attribute, which is not included in the exception
representation (and thus the logs).

Fixes internal issue 211616590
🤖 I have created a release *beep* *boop*
---


## [2.32.0](googleapis/python-bigquery@v2.31.0...v2.32.0) (2022-01-12)


### Features

* support authorized dataset entity ([googleapis#1075](googleapis#1075)) ([c098cd0](googleapis@c098cd0))


### Bug Fixes

* remove query text from exception message, use `exception.debug_message` instead ([googleapis#1105](googleapis#1105)) ([e23114c](googleapis@e23114c))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
* chore(deps): update all dependencies

* test: limit deps versions by python

* remove pyproj, as it is only a transitive dependency

Co-authored-by: Tim Swast <[email protected]>
Co-authored-by: Anthonios Partheniou <[email protected]>
renovate-bot and others added 23 commits April 17, 2023 15:51
… client (googleapis#1511)

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [x] Make sure to open an issue as a [feature request](https://togithub.com/googleapis/python-bigquery/issues/1512) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes 
- [feature request](https://togithub.com/googleapis/python-bigquery/issues/1512)🦕
- [internal bug](https://b.corp.google.com/issues/271044948)
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
…QueryJob (googleapis#1521)

* feat: expose configuration property on CopyJob, ExtractJob, LoadJob, QueryJob

Note for google-cloud-bigquery developers: This also refactors these classes so
that `_set_properties` does not modify the `_properties` dictionary in-place.
Doing so was also mutating the request object, making it difficult to debug
what request was _actually_ sent. Before this change, many tests hallucinated
that the request was always equal to the response.

* E           google.api_core.exceptions.BadRequest: 400 Clone operation with write disposition WRITE_TRUNCATE is not supported. Please try again with WRITE_EMPTY.
…gleapis#1527)

Source-Link: googleapis/synthtool@92006bb
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:2e247c7bf5154df7f98cce087a20ca7605e236340c7d6d1a14447e5c06791bd6

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…ibility issues (googleapis#1531)

* fix: loosen ipywidgets restrictions further to address ipython compatibility issues

* include ipywidgets in prerelease deps

* show all package versions

* add ipykernel dependency

* ipykernel in noxfile

* oops
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
* chore: update tests to be compatible with pandas 2.0

* use StringDtype without storage argument

* avoid Float64Dtype on older pandas
* chore: updates minimum version of bqstorage

* removes unneeded test

* updates linting, removes unneeded comment
)

* feat(bigquery): expose date_as_object parameter for users

* feat(bigquery): nit

* feat(bigquery): add unit test for date as object without pyarrow

* feat(bigquery): docs fixed

* feat(bigquery): nit

* feat(bigquery): nit
…google-resumable-media` (googleapis#770)

**Note**: PR is empty because this is purely to make sure CHANGELOG is updated. Follow-up to googleapis#768

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/python-bigquery/issues/new/choose) before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

Fixes googleapis#767  🦕
Release-As: 3.3.6

We're going to use 3.3.5 for a break-fix release.
@abdelmegahedgoogle abdelmegahedgoogle force-pushed the supportPartitionExpirationNoneValue branch from 1497090 to 90fc3fb Compare April 17, 2023 19:52
@abdelmegahedgoogle abdelmegahedgoogle requested a review from a team as a code owner April 17, 2023 19:52
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: xs Pull request size is extra small. labels Apr 17, 2023
@abdelmegahed
Copy link
Contributor

Re-opened this PR with an account that's approved for CLA. #1553

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.