Support for Consumer Grade CPUs?

[LuciAkirami]

Does Zendnn support for Consumer Grade AMD CPUs like Ryzen 5, 7 series having 8 cores or is the library specially designed to run on EPYC servers?

[vpirogov]

@LuciAkirami,

oneDNN, which this project is based on, supports broad range of x64 compatible processors, including AMD client and server CPUs.

[LuciAkirami]

Interesting. Thanks for the info @vpirogov , I will check this out

[LuciAkirami]

By the way, @vpirogov are there any benchmarks of running LLMs using oneDNN, any speed comparisons to native PyTorch, anything like that

[vpirogov]

Native Pytorch uses oneDNN for x64 and AArch64 optimizations.

oneDNN is a not a framework, it just provides optimized implementations of several performance critical functions, like matmul and convolution. It cannot run a model, but you can benchmark shapes you find important using matmul_perf example or benchdnn.

[LuciAkirami]

So if I'm not wrong, it provides optimised kernels, which even PyTorch uses, especially for the architectures you've mentioned

[vpirogov]

So if I'm not wrong, it provides optimised kernels, which even PyTorch uses, especially for the architectures you've mentioned

That's correct.

If you are looking for a framework to deploy LLMs on a PC consider OpenVINO. It is also using oneDNN optimized kernels and has fairly small footprint. You can find benchmarks, including some LLMs here.

[LuciAkirami]

Sure thanks for the suggestion, will check this out. I'm just getting started with learning cuda and writing simple kernels, so I checked this library out to maybe try out how optimisations work on cpu side.

Anyways I'll check both OpenVino and oneDNN so that I can get a bit deep into them.

[vpirogov]

Have fun!