Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standardising matrix arrays (adopt Array API spec) #941

Open
Tracked by #1791
KennethEnevoldsen opened this issue Jun 17, 2024 · 1 comment
Open
Tracked by #1791

Standardising matrix arrays (adopt Array API spec) #941

KennethEnevoldsen opened this issue Jun 17, 2024 · 1 comment
Labels
enhancement New feature or request v2 Issues and PRs related to `v2` branch
Milestone

Comments

@KennethEnevoldsen
Copy link
Contributor

Currently MTEB does not consistently use a ground truth for vector manipulation. A solution would be to adopt the Array API spec. A potentially simpler solution is to use either torch or numpy.

  • The benefit of using numpy is that it is very stable and rarely leads to OOM and related GPU issues.
  • The benefit of using PyTorch is that it potentially allows evaluation using GPUs (currently, sklearn does not support this, but might in the future, when they adopt they API spec)

partly related to the discussion in: #883

This is not a pressing concern atm.

@Muennighoff
Copy link
Contributor

Good point; I think using numpy would also allow us to make torch an optional dependency which could be nice as I assume some models do not require it so people have to install it just for mteb

@Samoed Samoed added the v2 Issues and PRs related to `v2` branch label Feb 6, 2025
@Samoed Samoed added this to the v2.0.0 milestone Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request v2 Issues and PRs related to `v2` branch
Projects
None yet
Development

No branches or pull requests

3 participants