Can you provide a script to test the decoding latency and peak memory usage?

[CR400AF-A]

Thanks for your great work!
I want to test the decoding latency and peak memory usage of these methods. I tried implementing it but the speed is slow than full kv.(Maybe I made some mistakes), so could you please provide the corresponding script as a reference?

[jinyucn]

Do you solve it? I also find that this method does not reduce the latency significantly.

[CR400AF-A]

Do you solve it? I also find that this method does not reduce the latency significantly.

Yes, I modified code and got results that seems to be correct. The reason is that GQA is not implemented properly in this repo(you can refer to the issue #49), so you will get lower results if you test on GQA-arch models.

Our code is not yet public, but I suggest that you can refer to the AdaKV repo, which implemented correct GQA. We also made changes based on their work.