[RTEB] qwen3-embedding-8b instruction-tuned for the corresponding retrieval tasks

[CyrilShch]

Hi! Congrats on a huge effort for releasing RTEB!
I was wondering if the results of qwen3-embedding-8b eval across the tasks are reported with its instruction-tuned version (since it's instruction-aware)?

If not, I'd like to suggest to include such since in my private benchmarks of code localization comparing the simple code instruction from the official qwen3-embedding repo could boost the recall@10 up to 10pp. Then, the whole picture of comparing top embedding models from oai / voyage (since they're not instruction-aware AFAIK) vs qwen3-embedding models can change drastically for specialized supported tasks.

The same for the gemini embedding model - it supports various tasks, specifying which can boost the retrieval quality by a margin

[Samoed]

Yes, we're evaluating qwen with instructions, but they can be a bit different #2907, because they haven't added them to implementation in our repo . gemmaembeding model also evaluated with instructions

[CyrilShch]

@Samoed Thx.

Yes, we're evaluating qwen with instructions, but they can be a bit different, because they haven't added them to implementation in our repo

Are those used instructions published in your repo somewhere? I'd nice to see them

[Samoed]

Some tasks specified in their Metadata

mteb/mteb/tasks/Clustering/eng/ArxivClusteringS2S.py

Line 41 in 12fe80b

prompt="Identify the main and secondary category of Arxiv papers based on the titles",

if tasks didn't specified it, then instructions from abs class are taken

mteb/mteb/abstasks/AbsTaskClustering.py

Line 65 in 12fe80b

abstask_prompt = "Identify categories in user passages."

[KennethEnevoldsen]

Since this is more of a question about the evaluation process, I will move it over to discussions (@Samoed outlines the current process nicely).