MTEB - how does leaderboard ranking work?

[brannair]

Hi there,

I plan to upload a new embedding model onto MTEB - but my model focusses just on the retrieval task so I only plan to upload this benchmark. How does Borda ranking work if we only upload 1 model task?

[KennethEnevoldsen]

For any score where the model is not evaluated, the model assumed gets a zero in the Borda count. Multiple models focus only on retrieval. I would still add it (because people might solve by the column), but you can also add it the the RTEB benchmark for retrieval:

[brannair]

That's great, thanks for the response. I'm not familiar with Borda rank, does this mean that the retrieval task rank = overall rank if that is the only task result that is provided? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Kenneth Enevoldsen ***@***.***> Sent: Wednesday, December 10, 2025 8:08:18 AM To: embeddings-benchmark/mteb ***@***.***> Cc: Nair, Brandon ***@***.***>; Author ***@***.***> Subject: Re: [embeddings-benchmark/mteb] MTEB - how does leaderboard ranking work? (Discussion #3706) For any score where the model is not evaluated, the model assumed gets a zero in the Borda count. Multiple models focus only on retrieval. I would still add it (because people might solve by the column), but you can also add it the the RTEB benchmark for retrieval: Screenshot.2025-12-10.at.14.07.35.png (view on web)<https://github.com/user-attachments/assets/852b5f9e-f1b4-4546-838f-682ccd4d1659> — Reply to this email directly, view it on GitHub<#3706 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/B3KLRWQTX62LAMGBKZLFGID4BALMFAVCNFSM6AAAAACOSPSHPCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMRSGAYDSMI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[brannair]

I re-read this, I think your statement actually means that overall rank is penalized for not having full scores. Can you confirm that?

[Samoed]

Yes, it is

[KennethEnevoldsen]

Yeah, but not penalizing would encourage people to submit only scores for tasks on which they performed well, which is not the incentive that we want to give.

This is an example of the places where we benchmark influences model development. By making this decision, we implicitly encourage general-purpose models over specialized models. This is the right direction to push the field in, but we do seek to diminish this influence by adding a specialized retrieval section to the leaderboard.

It is, of course, perfectly valid to disagree with this decision, and if so, we would love to hear what people have to say.