Created CoSENTLoss.py

[johneckberg]

PR Overview

• Resolve Implement CoSENT loss #2438

Details

• Implemented CoSENT loss as described in this blogpost and the AnglE repo.
• This loss follows the same input format as CosineSimilarityLoss, requiring each input to contain a pair of sentences and a floating point label, allowing for applications beyond two-category data. To use this loss with MNRLoss, like done here in the AnglE repo, an internal method must be used. I can't think of a way to use a different input format without assumptions about the labels, which would limit the applications beyond two-category data. If anyone has any ideas, take a look at the code and leave a comment!
• Subtraction by 1e^12 is how irrelevant values in the scores matrix become negligible in the logsumexp. I believe this is the constant used by the original implementation, and no other implementations have seen a need to change it. I agree, and for consistency have also used 1e^12.
• The example in the doc string uses InputExamples, not a Dataset/DatasetDict, which will need to change in the future.

[tomaarsen]

Hello!

I've ran various scripts that use CosineSimilarityLoss with both that loss & CoSENT, and the Spearman correlation coefficient for cosine similarity against the STSBenchmark test set almost universally shows improvements for CoSENT versus CosineSimilarityLoss.
This is very exciting! I'll be diving into the exact implementation details & blogpost soon.

• train_sts_indomain_semantic.py
  • Cosine: 84.21 Spearman correlation based on cosine similarity
  • CoSENT: 85.47 Spearman correlation based on cosine similarity
• training_multi-task.py
  • Cosine: 81.51 Spearman correlation based on cosine similarity
  • CoSENT: 82.41 Spearman correlation based on cosine similarity
• training_stsbenchmark_continue_training.py
  • Cosine: 86.35 Spearman correlation based on cosine similarity
  • CoSENT: 87.01 Spearman correlation based on cosine similarity
• training_stsbenchmark.py
  • Cosine: 79.18 Spearman correlation based on cosine similarity
  • CoSENT: 84.25 Spearman correlation based on cosine similarity

Code looks good too! Seems to match the other implementations.

As for your comments, it makes sense to consider this a drop-in replacement of the CosineSimilarityLoss, and thus support that input format (anchor-sentence pairs with a similarity score label). The 1e12 choice is also good - I'd like to stick to the official implementation.

• Tom Aarsen

[tomaarsen]

I think this might already be ready for merging! What do you think @johneckberg?

[johneckberg]

@tomaarsen Go ahead! If you think that the documentation is clear and easy enough to follow, then I also think it's ready for merging.

[tomaarsen]

I've added CoSENTLoss to the docs & updated the formatting slightly. I also mentioned my experiments where CoSENTLoss outperformed CosineSimilarityLoss.

---

cc: @bojone You might be interested in this work; we're impressed with the performance of CoSENT, and would like to implement it into Sentence Transformers to improve the accessibility of your loss function.

• Tom Aarsen

[bojone]

@tomaarsen Wonderfull! I'm very happy to see CoSENT receiving further recognition.

[tomaarsen]

Very excited to see this loss function merged. I'll be experimenting with it myself!