Report on pipeline and SPIR-V persistent cache implementation

[CLV-Iclucia]

I implemented persistent save/load APIs for Vulkan VkPipelineCache to enable disk serialization and reuse across application runs.
In addition, I designed and integrated a separate on-disk cache for compiled SPIR-V binaries, ensuring that redundant shader compilations are avoided and pipeline build efficiency is significantly improved across runs.

This function is still far from complete. Many problems remain to be solved.

Technical Learning

I read the documentation for APIs and guides of VkPipelineCache, and I read this blog post to learn about the best practice for storing, loading and validating pipeline cache.
I also learned the source code of NCNN to learn the whole process from compiling shaders to building the final compute pipeline, especially about current cache mechanism that computes the key of a pipeline to avoid repeated creation during a single run.

Changes Introduced

1. int PipelineCache::load_pipeline_cache(const char* path): This method will instruct PipelineCache object to load pipeline cache file from path. This method returns 0 if loading successfully and returns a nonzero value otherwise. If fails, PipelineCache will try to use empty VkPipelineCache object to create pipelines.
2. int PipelineCache::save_pipeline_cache(const char* path): This method will instruct PipelineCache object to save VkPipelineCache object as a file to path. This method returns 0 if saving the file successfully and returns a nonzero value otherwise.
3. void PipelineCache::set_shader_cache_dir(const char* dir): This method will set the SPIR-V code cache directory used by PipelineCache object to dir. All the SPIR-V code produced during creation will be saved under dir. When compiling shaders, PipelineCache will first try to look for file cache in the cache directory to skip compilation. If not specified, the default cache directory will be $LOCALAPPDATA/ncnn/shadercache on Windows and $HOME/.ncnn/shadercache on other platforms. Returns nonzero value if failing.
4. int PipelineCache::clear_shader_cache() const: This method will clear the current SPIR-V code cache directory. Returns nonzero value if failing.
5. Changes the signature of VulkanDevice::create_pipeline: add an argument of type VkPipelineCache* to enable creating VkPipeline using VkPipelineCache.
6. int VulkanDevice::create_empty_pipeline_cache(VkPipelineCache* vk_pipeline_cache): creates a VkPipelineCache object with empty data. Returns nonzero value if failing.
7. int VulkanDevice::create_pipeline_cache_with_data(const void* initial_data, size_t data_size, VkPipelineCache* vk_pipeline_cache): creates a VkPipelineCache object with initial data starting from initial_data with data_size bytes. Returns nonzero value if failing.
8. Add test test_pipeline_cache: this is a simple test for testing the functionality of pipeline cache.

Implementaion details:

I use vkGetPipelineCacheData to get the pipeline cache data binary and combine it with a file header for validation. The header format is

struct pipeline_cache_prefix_header
{
        uint32_t magic;
        uint32_t version; 
        uint32_t data_size; 
        uint32_t data_hash_fnv1a; // fnv1a hash

        uint32_t vendor_id;
        uint32_t device_id;
        uint32_t driver_version;
        uint32_t driver_abi; // sizeof(void*)

        uint8_t uuid[VK_UUID_SIZE];

        uint32_t reserved[4];
};

This design basically follows the practice in this blog post but adds version and reserved fields for possible future compatibility.
The design of spirv cache file is also like:

------------------------------
| Header | SPIR-V code binary |
------------------------------

The header for this is:

struct spv_cache_header
{
        uint32_t magic;          // magic number, 'SPVC' in host endian
        uint32_t header_version; // version of cache header format
        uint32_t ncnn_version;   // ncnn version when the cache is created

        uint32_t spv_size;          // size of spv binary data
        uint32_t data_hash_fnv1a;   // hash of spv binary data using fnv1a
        uint32_t data_hash_murmur3; // second hash of spv binary data using murmur3

        uint32_t vendor_id;
        uint32_t device_id;
        uint32_t driver_version;
        uint8_t uuid[VK_UUID_SIZE];
        uint32_t reserved[4]; // reserved for future use, must be zero
}

The design for this will be explained later.
When compiling a shader code, PipelineCache will first compute a key using multiple options. It will first use the key to search internal cache(using std::map). If this fails, it will use the decimal string of the key as file name to search for cache file in the cache directory. If succeed it will load the code and cache it in the internal cache.

Usage examples

int device_index = 0;
VulkanDevice* vkdev = vkdev->get_pipeline_cache();
int ret = vkdev->get_pipeline_cache()->load_pipeline_cache("./vk_pipeline_cache");
if (ret == 0) 
{
    // load successfully and all the pipeline creations will use the cache loaded
}
else
{
   // start with an empty pipeline cache to create the pipelines
}
int ret = vkdev->get_pipeline_cache()->save_pipeline_cache("./vk_pipeline_cache");
if (ret == 0) 
{
    // succeed
}
else
{
   // fail
}

Problems and solutions

1. Bottleneck of pipeline creation

I found that simply using VkPipelineCache can accelerate vkCreatePipeline greatly, but the cost of this step seems unsignificant compared with shader compilation.

So I have to implement shader SPIR-V cache to accelerate this process.

2. Cross-platform file operations

The project currently uses C++11, which does not provide a unified API for filesystem operations such as renaming or removing files. As a result, platform-specific implementations were required inside pipelinecache.cpp.

For now, I implemented platform-specific handling directly in pipelinecache.cpp for minimal changes. A possible future improvement would be to abstract these into a dedicated cross-platform file utility module (similar to how some projects adopt a filesystem.h wrapper).

3. SPIR-V cache invalidation strategy

The content of compiled SPIR-V binaries can change due to multiple factors:

• shader source modifications
• GLSLang compiler version differences
• NCNN handling of device extensions and pipeline options

If these are not accounted for, stale SPIR-V caches may cause incorrect or incompatible pipelines.

I introduced an ncnn_version field in the SPIR-V cache header. This field should be updated whenever relevant changes are introduced (e.g., new GLSLang versions or internal NCNN changes affecting shader compilation). On load, the version is validated, and outdated caches are discarded. But I believe this is not the best practice. Perhaps updating this field automatically in the building system is better.

4. Testing and API exposure

There is a tradeoff between providing flexible testing APIs for SPIR-V cache and keeping the public API surface minimal. Exposing too many low-level cache file operations complicates the API, while hiding them makes unit testing difficult.

The only thing I can do is use two hashes but that is far from enough.

5. Security of SPIR-V cache files

If a SPIR-V cache file is maliciously altered, and both the content and the hash are manipulated, the cache may load compromised shaders and use them to create false pipelines.

The only thing I can do is use multiple hash codes but that is far from enough.

Performance

In a single pipeline creation test, the time taken for creating a pipeline is reduced from 90ms to 0.4ms using the two caches across runs (mocked by creating and destroying GPU repeatedly) on my PC and this is mainly contributed by spirv code cache.

The CPU is AMD Ryzen 7 5800H and the GPU is Nvidia RTX 3060.

pipeline cache test creation time (without cache): 90.87 ms
pipeline cache test creation time (with cache): 0.37 ms

The main contribution comes from SPIR-V cache.