Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(test): Force usage of ReadAPI #3625

Merged
merged 1 commit into from
Jan 9, 2025

Conversation

o-shevchenko
Copy link
Contributor

@o-shevchenko o-shevchenko commented Jan 7, 2025

Fixes googleapis/java-bigquerystorage#2764

Related issue: #3574

SDK still uses the old API and just skips ReadAPI, even if it's enabled.
This happens because of this check:

In order to use ReadAPI, we need to add any of these configurations, for example, setJobTimeoutMs.
After this, we will see Not Using Fast Query Path, using jobs.insert and Using BigQuery Read API instead of Using Fast Query Path.

After enabling ReadAPI we've got 6-10x speed improvement:

Benchmark                                     (rowLimit)  Mode  Cnt       Score   Error  Units
ConnImplBenchmark.iterateRecordsUsingReadAPI      500000  avgt        16439.884          ms/op
ConnImplBenchmark.iterateRecordsUsingReadAPI     1000000  avgt        16005.065          ms/op
ConnImplBenchmark.iterateRecordsUsingReadAPI    10000000  avgt        74471.796          ms/op
ConnImplBenchmark.iterateRecordsUsingReadAPI    50000000  avgt       484732.516          ms/op
ConnImplBenchmark.iterateRecordsUsingReadAPI   100000000  avgt        48608.101          ms/op

@o-shevchenko o-shevchenko requested a review from a team as a code owner January 7, 2025 14:26
@o-shevchenko o-shevchenko requested a review from tswast January 7, 2025 14:26
@product-auto-label product-auto-label bot added size: s Pull request size is small. api: bigquery Issues related to the googleapis/java-bigquery API. labels Jan 7, 2025
@o-shevchenko o-shevchenko changed the title Force usage of ReadAPI fix(test): Force usage of ReadAPI Jan 7, 2025
@PhongChuong
Copy link
Contributor

/gcbrun

@PhongChuong
Copy link
Contributor

Thanks for the fix.

@tswast tswast removed their request for review January 7, 2025 16:36
@PhongChuong PhongChuong added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Jan 9, 2025
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jan 9, 2025
@PhongChuong PhongChuong merged commit 5ca7d4a into googleapis:main Jan 9, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/java-bigquery API. size: s Pull request size is small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reading data using the executeSelect API is slow
3 participants