feat: Cleaned up scripts folder #1144
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- A few of the benchmarks weren't implemented in the benchmark list. The best choice seem to move them benchmarks and remove the script (there were already a few issues). To ensure that they are maintained with the rest. - Removed trivial (e.g. running one task) or outdated examples. Removing them ensure that we have less to maintain. - moved mteb specific script into a mteb-folder.
2 tasks
KennethEnevoldsen
changed the title
Muennighoff
approved these changes
Aug 11, 2024
There was a problem hiding this comment.
Nice! Rather than breaking it up into mteb & mmteb, I would probably just break it up into scripts/data, scripts/running and maybe scripts/analysis and put single files directly into scripts rather than having folders with just one file 🤔
But I think the current way is also fine if you prefer!
This avoids loading in classes as modules (see PR ...)
isaac-chung
approved these changes
Aug 11, 2024
|
That is perfectly fine with me as well. I have changed it to match |
isaac-chung
added a commit
that referenced
this pull request
Aug 14, 2024
* fix: when create meta merge results with existing README.md (#1117) * init version * merge scores * add test without frontmatter * lint * 1.12.91 Automatically generated by python-semantic-release * fix: IWSLT2017BitextMining loading dataset. (#1132) fix: the way of loading and transform dataset and typo of filename on IWSLT2017BitextMining * 1.12.92 Automatically generated by python-semantic-release * fix: Allow more linient TaskMetadata (#1131) * fix: Allow more linient TaskMetadata This fix allows for more lenient TaskMetadata when using the package, but maintain the validation for the package within the tests. This is intended to ensure flexibility and ease of use, while maintaining high-quality doucementation of tasks * Added fixes based on corrections * Update points table * 1.12.93 Automatically generated by python-semantic-release * Added pyupgrade (using ruff) to lint CI (#1137) * Added pyupgrade (using ruff) to lint CI * Added FA ruleset * Added C4 ruleset for simplifying comprehensions * Added a few exceptions for points Co-authored-by: bryant1410 <[email protected]> * Added required import * Add co-author Co-authored-by: bryant1410 <[email protected]> * format --------- Co-authored-by: bryant1410 <[email protected]> Co-authored-by: bryant1410 <[email protected]> * fix: add CoIR tasks (#1130) * add CoIR tasks * change to mteb datasets * Update points table * Update tasks table * 1.12.94 Automatically generated by python-semantic-release * feat: Added in functionality to allow loading outdated results (#1141) * 1.13.0 Automatically generated by python-semantic-release * fix: add CoIR as Benchmark (#1142) * add CoIR as Benchmark * lint * 1.13.1 Automatically generated by python-semantic-release * Simplify models (#1118) * Merge * Adapt * Simplify * Check for rev again * Rmv cmmnt * Simplify * simplify * Rmv comment Co-authored-by: Kenneth Enevoldsen <[email protected]> * Use logging; change try except; add info * Lint * Rmv results * Update rev * Simplify models; Allow instructions * Jobs * Fix merge * Format * Adapt models * Fix task types * Update * Fix syntax * Simplify * Add comparison * Format * Fix double comment Co-authored-by: Kenneth Enevoldsen <[email protected]> * Move example * Format * Rmv outdated instructions --------- Co-authored-by: Kenneth Enevoldsen <[email protected]> * Missing import for SadeemQuestionRetrieval (#1146) Fixes #1143 * fix: Remove unused tests (#1148) These tests are no longer used since results moved to the results repo. * Update tasks table * 1.13.2 Automatically generated by python-semantic-release * feat: Cleaned up scripts folder (#1144) * fix: Move benchmarks from script to benchmarks.py - A few of the benchmarks weren't implemented in the benchmark list. The best choice seem to move them benchmarks and remove the script (there were already a few issues). To ensure that they are maintained with the rest. - Removed trivial (e.g. running one task) or outdated examples. Removing them ensure that we have less to maintain. - moved mteb specific script into a mteb-folder. * format * fix: Convert arbitrary imports to absolute imports. (#1145) This avoids loading in classes as modules (see PR ...) * restructe scripts folder * 1.14.0 Automatically generated by python-semantic-release * Fix: llm2vec for retrieval (#1152) * fix: remove kwargs LLM2Vec doesn't have * fix: linter error * fix: an item automatically added by linter * fix: removed test set for AFQMC with no gold labels (#1153) The test set of AFQMC is hidden so has removed it from the default suite. This influences the C-MTEB. Who is the best person to contact for this? (@Muennighoff) * 1.14.1 Automatically generated by python-semantic-release * LitSearchRetrieval added (#1150) * LitSearchRetrieval added * points added * fixed descriptive_stats in LitSearchRetrieval * fix lint * metadata fixed for LitSearchRetrieval * metadata fixed for LitSearchRetrieval again * metadata fixed for LitSearchRetrieval again * points 1150 updated * metadata fixed for LitSearchRetrieval again * lint fix * Update points table * Update tasks table * ruff check fixed * make lint * fix imports * make lint --------- Co-authored-by: Roman Solomatin <[email protected]> Co-authored-by: github-actions <[email protected]> Co-authored-by: Hiroki Iida <[email protected]> Co-authored-by: Kenneth Enevoldsen <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Roman Solomatin <[email protected]> Co-authored-by: Niklas Muennighoff <[email protected]> Co-authored-by: dokato <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This caused a change in the order of imports, which turns out that (due to our shady naming practices) causes:
to import the Module
TaskMetadatainstead of the object. I have instead specified all places to avoid future issues:I have moved this fix to the associated PR: #1145 to clarify it.
Checklist
make test.make lint.