feat: embedding distance calculator

[marieaurore123]

• I have looked for existing issues (including closed) about this

Feature Request

Add feature to rig which can help users measure semantic similarity (or dissimilarity) between a prediction and a reference string.

Motivation

This feature is very related to LLMs and Embeddings and could be useful both Rig itself (improving the in-memory vector store) and for other projects that depend on Rig.

Proposal

pub enum DistanceMetric {
    Cosine,
    L2,
    Dot,
    Manhattan,
}

impl rig::embeddings::embedding::Embedding {
    fn cosine_dist(embedding: Embedding) -> f64;
    fn dot_dist(embedding: Embedding) -> f64;
    fn l2_dist(embedding: Embedding) -> f64;
    fn manhattan_dist(embedding: Embedding) -> f64;
}

• Update in memory vector store with this feature
• Set implementation behind feature flag.
• Maybe - instead of adding semanticsimilarity_rs, copy only necessary code

Dependencies:
semanticsimilarity_rs

Alternatives

[cvauclair]

This makes a lot of sense! I think it makes more sense to have it part of rig-core directly since it's closely related to the existing embedding API. However, depending on how "heavy" the dependency is (it does depend on Rayon), this feature could be behind a feature flag

[cvauclair]

Alternatively, what could be interesting for both flexibility and completeness would be to rewrite the distance algorithms from semanticsimilarity_rs without using Rayon, and have rayon as a feature flag which would overwrite those implementations with the ones from semanticsimilarity_rs (or our own versions which uses rayon).

This can be done like using something like:

struct Embedding {...}

#[cfg(feature = "rayon")]
impl Embedding {
  pub fn cosine_distance(&self, other: &Embedding) -> f64 {
    // implementation using rayon or semanticsimilarity_rs
  }
}

#[cfg(not(feature = "rayon"))]
impl Embedding {
  pub fn cosine_distance(&self, other: &Embedding) -> f64 {
    // implementation with stdlib (without rayon or semanticsimilarity_rs)
  }
}

[cvauclair]

Closed by #142