advice for getting to this level of expertise at writing CUDA

[j93hahn]

Hi @siboehm, thank you for this blog post. It's an absolute gem to read through and I gained a tremendous amount of insight. I've been writing cuda for ~6 months now, and wanted to ask how you were able to get to this level? for example, your knowledge of the compiler, warp-level manipulation, and many other things that I thought were NVIDIA trade secrets [e.g., their entire cuBLAS library]

is it a lot of reading the CUDA documentation and simply practicing writing CUDA kernels? it seems near-impossible to catch up to the best kernel engineers today; any insight would be greatly appreciated, thank you!

[Infatoshi]

hows the CUDA journey going?
maybe start with freecodecamp's CUDA course on youtube. its a good enough start to show you whats possible and it will likely become easier to narrow down the point of optimizations so you dont have to rely on someone else at anthropic or nvidia.