Mteb Versions Save

MTEB: Massive Text Embedding Benchmark

1.12.27

2 weeks ago

1.12.27 (2024-06-13)

Documentation

  • docs: Add points for #911 (#913)

Add poinhts (5deeb3c)

Fix

  • fix: Update annotations for English STS tasks (#908)

  • Update STS16STS.py

  • Update STS15STS.py

  • Update STS14STS.py

  • Update STS13STS.py

  • add points for 902&908 prs

  • Update STS16STS.py

  • Update STS14STS.py

  • Update STS14STS.py

  • Update STS16STS.py

Co-authored-by: Tikhonova Maria <[email protected]> (f1dd8bb)

Unknown

1.12.26

3 weeks ago

1.12.26 (2024-06-13)

Documentation

  • docs: Update annotations for multilingual STS tasks (#902)

  • Update STS12STS.py

  • Update STSBenchmarkMultilingualSTS.py

  • Update STS22CrosslingualSTS.py

  • Update STS17CrosslingualSTS.py

  • Update STS22CrosslingualSTS.py

  • Update STSBenchmarkMultilingualSTS.py

  • Update STS17CrosslingualSTS.py

  • Update STS12STS.py

  • Update STS17CrosslingualSTS.py

  • Update STSBenchmarkMultilingualSTS.py

  • Update STS22CrosslingualSTS.py

Co-authored-by: Tikhonova Maria <[email protected]> (4383fd3)

Fix

  • fix: Incorrect handling of qrel_revision fix: #909 (#911)

  • fix: #909

  • fix: #909

  • add: points

  • clean (63cd4b7)

Unknown

  • Update tasks table (30433a9)

  • Update points table (d771437)

  • (1) Add WebLINX Candidate Elements Reranking Task; (2) move convert_conv_history_to_query from mteb.evaluation.evaluators.RetrievalEvaluator to mteb.evaluation.evaluators.utils (#820)

  • Initial file (WIP)

  • Update dates, and indicate dotos

  • Update mteb/tasks/Reranking/eng/WebLINXCandidatesReranking.py

  • Move convert_conv_history_to_query from RetrievalEvaluator to utils

  • Add WebLINXCandidatesReranking to mteb.tasks.Reranking

  • Lint RetrievalEvaluator's imports

  • lint mteb.evaluation.evaluator.utils

  • Update dataset path, name and revision, n_samples and avg_character_lenght

  • Update revision, add load_data method

  • Fix typo

  • Change main score to mrr, since recall@10 is not supported

  • Add average results

  • Add points

  • Run make lint (e34ddaa)

  • Update tasks table (b16d7d5)

1.12.25

3 weeks ago

1.12.25 (2024-06-11)

Documentation

  • docs: Add points for paper writing (#901)

  • add points for paper writing

  • add PR number

  • correct typo (fbbc44b)

Fix

  • fix: Backfilled bibtex citations data (#900)

  • Added IndicNLP News Classificaiton

  • Added IndicNLP News Classificaiton

  • Added results

  • Updated dataset version

  • Small fixes

  • Small fix

  • Small fix

  • Updated results

  • Fix linting issues

  • Added points

  • Resolve conflict

  • Update 610.jsonl

  • Backfilled missing bibtex citations.

  • Backfilled missing bibtex citations.

  • Remove non-present files

Co-authored-by: Imene Kerboua <[email protected]> Co-authored-by: Kenneth Enevoldsen <[email protected]> (77d0e06)

Unknown

1.12.24

3 weeks ago

1.12.24 (2024-06-09)

Fix

  • fix: Add openai and voyage models (#887)

  • Add openai and voyage models

  • Add rate limit for voyage model

  • Add rate and token limit for voyage (ad9b3ce)

Unknown

1.12.23

3 weeks ago

1.12.23 (2024-06-08)

Documentation

  • docs: Update points.md (#890) (4318c82)

Fix

  • fix: abstention metric for small datasets (#893)

  • fix abstention bug

  • make lint (9d28296)

Unknown

  • Update tasks table (bab7503)

  • Update points table (c61ab09)

  • Added multilabel stratification to AbsTaskMultilabelClassification (#760)

merge conflicts fixed for stratification (d7dc9a8)

1.12.22

4 weeks ago

1.12.22 (2024-06-06)

Fix

  • fix: Add GritLM (#880)

  • Add GritLM

  • Remove unused imports

  • Use embedding mode to save memory

  • Format

  • Format

  • Add langs

Co-authored-by: Kenneth Enevoldsen <[email protected]>

  • Add langs

Co-authored-by: Kenneth Enevoldsen <[email protected]>

  • Change loader

Co-authored-by: Kenneth Enevoldsen <[email protected]> (0c99a4e)

1.12.21

4 weeks ago

1.12.21 (2024-06-05)

Documentation

  • docs: Added source for CmedqaRetrieval (#886)

docs: update CmedqaRetrieval description to specify source (3e910ff)

Fix

  • fix: Add error reporting for Retrieval (#873)

  • add error reporting.

  • no message

  • Update mteb/abstasks/AbsTaskRetrieval.py Co-authored-by: Isaac Chung <[email protected]>

  • add kwarg args

  • change to metadata.name

  • fix format

  • Update mteb/abstasks/AbsTaskRetrieval.py Co-authored-by: Isaac Chung <[email protected]>

  • change to explicit args

  • remove cmdline changes

Co-authored-by: Isaac Chung <[email protected]> (5397bd2)

1.12.20

4 weeks ago

1.12.20 (2024-06-05)

Fix

  • fix: Updated CLI for MTEB (#882)

  • Updated CLI for MTEB

It now includes three main commands one for running, one for getting an overview and one for creating the metadata for hf.

I also

  • added lower bound on dependencies as it caused a few issues.
  • deleted some results with a model revision attached (also created a fix to make sure that doesn't happen as much going forward)
  • Added a test for the cli
  • Added a quite extensive docstring to the CLI
  • made relevant changed to the documentation

  • ensure tests pass

  • minor changes to PR template

  • Minor changes to PR template

  • remove tmp file

  • fixed failing test and updated it to avoid future false positives

  • fix deprecation warning for logger.warn

  • don't clear path before running tests as it disturbs other tests when run in parallel (c6f618b)

1.12.19

4 weeks ago

1.12.19 (2024-06-05)

Documentation

  • docs: minor fix for point validation to avoid error when people split up points (7d0d631)

Fix

  • fix: Add CEDR, SensitiveTopics for multilabel and RuBQ for reranking (#881)

  • add russian reranking and multilabel tasks

  • fix import order

  • add results for baselines

  • add points

  • add points for review (9128df4)

Unknown

It now includes three main commands one for running, one for getting an overview and one for creating the metadata for hf.

I also

  • added lower bound on dependencies as it caused a few issues.
  • deleted some results with a model revision attached (also created a fix to make sure that doesn't happen as much going forward)
  • Added a test for the cli
  • Added a quite extensive docstring to the CLI
  • made relevant changed to the documentation (29b1c34)

1.12.18

4 weeks ago

1.12.18 (2024-06-05)

Fix

  • fix: Ensure result are consistently stored in the same way (#876)

  • Ensure result are consistently stored in the same way

  • (due to failing test): updated missing dataset references
  • (to test with more than one model) Added e5 models base and large
  • updated mteb.get_model to now include metadata in the model object
  • ensure that model name is always included when saving (with a default when it is not available)
  • use the ModelMeta for the model_meta.json

  • format

  • minor test fixes

  • docs: Minor updated to repro. workflow docs

  • fixed failing test

  • format

  • Apply suggestions from code review

Co-authored-by: Isaac Chung <[email protected]>

  • docs: update PR template

  • fix: Added benchmark object (#878)

  • removed duplicate task

  • Added benchmark object

  • removed import for duplicate task

  • fix dataset references

  • added seb

  • Added test for running benchmarks

  • changed tasks to be an iterable

  • format

  • Apply suggestions from code review

Co-authored-by: Niklas Muennighoff <[email protected]> Co-authored-by: Isaac Chung <[email protected]>

Co-authored-by: Isaac Chung <[email protected]> Co-authored-by: Niklas Muennighoff <[email protected]> (fb843d0)

Unknown