Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some BEIR queries are unused #2575

Closed
lintool opened this issue Aug 22, 2024 · 1 comment
Closed

Some BEIR queries are unused #2575

lintool opened this issue Aug 22, 2024 · 1 comment

Comments

@lintool
Copy link
Member

lintool commented Aug 22, 2024

E.g.,

% gunzip -c topics.beir-v1.0.0-bioasq.test.bge-base-en-v1.5.jsonl.gz | wc
gunzip -c topics.beir-v1.0.0-bioasq.test.splade-pp-ed.tsv.gz | wc
gunzip -c topics.beir-v1.0.0-bioasq.test.splade_distil_cocodenser_medium.tsv.gz | wc
gunzip -c topics.beir-v1.0.0-bioasq.test.tsv.gz | wc
gunzip -c topics.beir-v1.0.0-bioasq.test.unicoil-noexp.tsv.gz | wc
gunzip -c topics.beir-v1.0.0-bioasq.test.wp.tsv.gz | wc

    3743 2885853 63739253
     500 1185261 7980762
     500 1050806 6911775
     500    4525   39190
     500  401383 2265529
     500    7037   44962

This means retrieval is wasting cycles generating results that won't be evaluated.
@Panizghi Panizghi mentioned this issue Sep 10, 2024
Merged
@lintool
Copy link
Member Author

lintool commented Sep 15, 2024

Closed by castorini/anserini-tools#83

@lintool lintool closed this as completed Sep 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@lintool