-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed document source and score field mismatch in sorted hybrid queries #1043
Merged
martin-gaievski
merged 3 commits into
opensearch-project:main
from
martin-gaievski:fixed_mismatch_in_doc_source_and_score_fields
Jan 3, 2025
Merged
Fixed document source and score field mismatch in sorted hybrid queries #1043
martin-gaievski
merged 3 commits into
opensearch-project:main
from
martin-gaievski:fixed_mismatch_in_doc_source_and_score_fields
Jan 3, 2025
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
martin-gaievski
force-pushed
the
fixed_mismatch_in_doc_source_and_score_fields
branch
from
December 25, 2024 00:53
093f199
to
3559f12
Compare
martin-gaievski
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
sean-zheng-amazon,
model-collapse,
zane-neo,
vibrantvarun,
zhichao-aws,
yuye-aws and
minalsha
as code owners
December 25, 2024 00:55
martin-gaievski
added
backport 2.x
Label will add auto workflow to backport PR to 2.x branch
Bug Fixes
Changes to a system or product designed to handle a programming bug/glitch
labels
Dec 25, 2024
ryanbogan
approved these changes
Dec 25, 2024
src/main/java/org/opensearch/neuralsearch/search/collector/HybridTopFieldDocSortCollector.java
Show resolved
Hide resolved
src/main/java/org/opensearch/neuralsearch/search/collector/HybridTopFieldDocSortCollector.java
Outdated
Show resolved
Hide resolved
…is enabled in hybrid query Signed-off-by: Martin Gaievski <[email protected]>
Fixed mismatch between document source and score fields when sorting is enabled in hybrid query Signed-off-by: Martin Gaievski <[email protected]>
Signed-off-by: Martin Gaievski <[email protected]>
martin-gaievski
force-pushed
the
fixed_mismatch_in_doc_source_and_score_fields
branch
from
January 3, 2025 21:27
c46ae50
to
f3f177c
Compare
LGTM Thanks. |
vibrantvarun
approved these changes
Jan 3, 2025
junqiu-lei
approved these changes
Jan 3, 2025
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jan 3, 2025
…es (#1043) * Fixed mismatch between document source and score fields when sorting is enabled in hybrid query Signed-off-by: Martin Gaievski <[email protected]> (cherry picked from commit 030e3f4)
martin-gaievski
added a commit
to martin-gaievski/neural-search
that referenced
this pull request
Jan 3, 2025
…es (opensearch-project#1043) * Fixed mismatch between document source and score fields when sorting is enabled in hybrid query Signed-off-by: Martin Gaievski <[email protected]> (cherry picked from commit 030e3f4) Signed-off-by: Martin Gaievski <[email protected]>
martin-gaievski
added a commit
that referenced
this pull request
Jan 3, 2025
…es (#1043) (#1057) * Fixed mismatch between document source and score fields when sorting is enabled in hybrid query (cherry picked from commit 030e3f4) Signed-off-by: Martin Gaievski <[email protected]> Co-authored-by: Martin Gaievski <[email protected]>
heemin32
pushed a commit
to heemin32/neural-search
that referenced
this pull request
Jan 9, 2025
…es (opensearch-project#1043) (opensearch-project#1057) * Fixed mismatch between document source and score fields when sorting is enabled in hybrid query (cherry picked from commit 030e3f4) Signed-off-by: Martin Gaievski <[email protected]> Co-authored-by: Martin Gaievski <[email protected]>
martin-gaievski
added a commit
that referenced
this pull request
Jan 10, 2025
…es (#1043) * Fixed mismatch between document source and score fields when sorting is enabled in hybrid query Signed-off-by: Martin Gaievski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backport 2.x
Label will add auto workflow to backport PR to 2.x branch
Bug Fixes
Changes to a system or product designed to handle a programming bug/glitch
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed document source and score field mismatch in sorted hybrid queries.
Returned search hits identified with the help of min heap, it's used by sorting functionality to get top X docs. We do keep track of the heap leaf element and updating it when collecting doc ids. In current code we use only one element for this, but in case of hybrid query we do need separate element for each sub-query. With a single element our updates are incorrectly propagated to a results of different sub-queries, cause different types of inconsistency: doc id, score fields, score can be incorrect in final search result.
In this PR I'm changing the min heap leaf element from a single object to an array of objects, one for each sub-query.
Tested on the data set referred in the issue, got correct response where all field have consistent values in
_source
andsort
sections:Related Issues
#1044
Check List
[ ] New functionality has been documented.[ ] API changes companion pull request created.--signoff
.~~[ ] Public documentation issue/PR created.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.