fix(bigquery): Remove ReadAPI bypass in executeSelect() #3624
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
product-auto-label
bot
added
size: m
Move readAPI condition check from getExecuteSelectResponse() to queryRpc(). This allows fast query to be used with ReadAPI.
product-auto-label
bot
added
size: s
whuffman36
requested review from
mrfaizal,
shollyman and
PhongChuong
and removed request for
mrfaizal
|
Still working on ideas for a testing strategy to make sure the requests are going to the endpoints that we want, but I wanted to get your opinions on the fix. |
A Boolean field is added to keep track of whether or not the high throughput ReadAPI is used. This is mostly for testing to avoid another regression in the future.
product-auto-label
bot
added
size: m
PhongChuong
reviewed
Jan 14, 2025
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/JobStatistics.java
Show resolved
Hide resolved
| @@ -476,22 +476,29 @@ private BigQueryResult queryRpc( | |||
| } | |||
|
|
|||
| // Query finished running and we can paginate all the results | |||
| if (results.getJobComplete() && results.getSchema() != null) { | |||
| // Results should be read using the high throughput read API if sufficiently large. | |||
| boolean shouldUseReadApi = | |||
There was a problem hiding this comment.
nit: shouldUseReadApi is very similar to the method: useReadAPI which is used in getSubsequentQueryResultsWithJob. In addition, even when shouldUseReadApi == true here, the useReadAPI() can return false in the subsequent getSubsequentQueryResultsWithJob call. Perhaps, changing the name to a negative condition is sufficient. E.g., shouldNotUseReadApi (or something else better).
There was a problem hiding this comment.
What do you think of resultsLargeEnoughForReadApi?
PhongChuong
approved these changes
Jan 15, 2025
This was referenced Feb 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the ReadAPI is being bypassed in the
executeSelect()function when requests are sent using fast query. This change decides whether to use the ReadAPI based on the query result size, and enables the use of both fast query and ReadAPI together.This change also updates
queryRpc()to use the table data from the first page of the query results instead of the empty results of the unfinished job.Finally, this change adds a subfield in the
QueryStatisticsfield in the query response to keep track of whether or not the ReadAPI is used. This field is essentially only useful for testing to ensure that another regression is avoided in the future.Fixes #2764