Limit results generated by SELECT queries in MSQ #14370
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adarshsanjeev
approved these changes
Jun 7, 2023
cryptoe
reviewed
Jun 7, 2023
| return Sequences.simple(retVal); | ||
| } | ||
| ) | ||
| .limit(Limits.MAX_SELECT_RESULT_ROWS) |
There was a problem hiding this comment.
For the end user, we need to set a flag in the report mentioning the results are truncated.
cryptoe
reviewed
Jun 9, 2023
| ).collect(Collectors.toList()); | ||
|
|
||
| final List<SqlTypeName> sqlTypeNames = task.getSqlTypeNames(); | ||
| final List<Object[]> retVal = new ArrayList<>(); |
There was a problem hiding this comment.
This might blow up.
A good test case for this issue is to try to do a select * on a datasource with 1 million rows.
|
Please add the truncated flag in the read me. |
cryptoe
approved these changes
Jun 13, 2023
cryptoe
added
the
Area - MSQ
|
cc @vogievetsky We now have a new field inside the task report : |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
MSQ queries return the results in the query reports. Querying large data sources can generate a lot of rows, all of which will get materialized in the query reports, causing them to grow very large. This PR limits the results of the SELECT queries that we put in the reports.
Note: Only way to fetch the complete result set after this change would be to run the query using the native engine, however, we might add a way to execute long-running queries that post their complete results in a different. Therefore the rows in the reports would serve as a preview to the actual result set.
Release note
SELECT queries executed using MSQ generate only a subset of the results in the query reports.
Key changed/added classes in this PR
ControllerImplThis PR has: