Improving input token throughput?

[carlesoctav]

so, i ran some benchmark with VLLM (I actually modified some parts of the code so it would work with vllm on tpu), and the logs show something like this:

Processed prompts:  16%|▏| 1456/9263 [37:28<11:01:04,  5.08s/it, est. speed input: 86.99 toks/s, outp                                                                                                     ProceProcessed prompts:  64%|████████████████████████████████████████████████████████                               

| 5971/9263 [2:16:18<1:17:49,  1.42s/it, est. speed input: 72.62 toks/s, output: 2683.22 toks/s]

the input tok/s is much worse compared to the output tok/s, do you think this could be a bottleneck?

[NathanHB]

hey ! so yes this is probably the bottleneck. What task are you using ?
Pretty sure it's a vLLM issue though, as we feed the engine with all the inputs in one time and let the engine process them.

[carlesoctav]

this one: https://github.com/carlesoctav/lighteval/blob/1b6b5ccff13365ef2d5b28ad6f027cb0ca80c9d9/eval.py
i also made some change since vllm-tpu is using vllm 0.11
https://github.com/carlesoctav/lighteval/blob/1b6b5ccff13365ef2d5b28ad6f027cb0ca80c9d9/src/lighteval/models/vllm/vllm_model.py#L464

[NathanHB]

alright ! And do you need any help on this ? I dont think the bottleneck is caused by lighteval but if you think otherwise we would love to work towards making it better

[carlesoctav]

nah, all good. im opening this issue probbaly to aks for a comparison of vllm input throughput on GPU.