CUDA (nvcc) caching

Kache caches single-source nvcc -c and nvcc -dc object compilations. If it cannot prove that an invocation is safe to cache, it runs the real compiler unchanged.

Setup

For CMake:

export CUDACXX="kache nvcc"
# or, non-invasively, as a launcher:
cmake -DCMAKE_CUDA_COMPILER_LAUNCHER=kache ...

For Make and build scripts that honor NVCC:

export NVCC="kache nvcc"

Kache recognizes the nvcc command with or without a path prefix, including nvcc.exe.

What is cached

Kache caches the single-source -c device compile as a whole:

nvcc -c src/kernel.cu -o build/kernel.o -arch=sm_80

For separate compilation, -dc and -c -rdc=true also cache the object. Device linking and final linking run through nvcc uncached:

nvcc -dc src/kernel.cu -o build/kernel.o
nvcc -dlink build/kernel.o build/helper.o -o build/device-link.o

nvcc is a driver: one -c invocation fans out to the device frontend, one compiler pair per GPU architecture, the fatbinary bundler, and the host C++ compiler. Kache caches the whole invocation as one entry. On a miss it runs nvcc end to end and stores the final object, unless a file the key hashed was modified while the build ran, or just before it by the margin file times can trail the clock: within the same kernel clock tick on Linux, 1 ms on macOS and Windows, 20 ms on other systems, and 2 seconds on file systems that keep whole seconds, such as HFS+ and FAT. A timestamp that is in the future when Kache reads the file counts as clock skew, not a change. On a hit it restores the object (and the -MF dep-info sidecar, when requested) and skips the toolchain.

Why the key is sound

nvcc -E is not a safe cache key: it always defines __CUDA_ARCH__, so host-only code guarded by #ifndef __CUDA_ARCH__ is invisible in its output. Kache does not hash preprocessed output. The key is built from:

• nvcc's version and the host compiler's version (a host upgrade changes the object, probed via nvcc --dryrun);
• the modeled flags verbatim: -dc / -rdc mode, GPU -arch / -gencode / --generate-code / -code, the host compiler (-ccbin / --compiler-bindir), -std, -O, defines, and the -Xcompiler / -Xptxas / -forward-unknown-to-host-compiler forwarding flags;
• the dependency closure: the raw content of the source plus every header nvcc -M reports. Hashing raw contents (not preprocessed output) captures host-only code, so any source or header edit invalidates the entry.
• the driver environment: NVCC_PREPEND_FLAGS / NVCC_APPEND_FLAGS are folded in verbatim, so the same checkout with different driver flags keys apart.

Remote sharing

On a miss Kache adds host -ffile-prefix-map rules through -Xcompiler, so the object does not name the checkout. The rules map the build directory, the source's directory, the output's directory, and any [paths] base_dirs roots to fixed names, and header paths in the key use the same names. KACHE_BASE_DIR does not apply to nvcc.

Flag values are keyed as written, so an absolute path inside a flag, such as -I/home/me/project/include or an absolute -MF path, gives each checkout its own key. Use relative paths in flags, or build from the same absolute path, to share entries. If an object still contains one of the mapped roots, Kache uses it for that build but does not store it. KACHE_NVCC_PATH_NORMALIZE=0 turns path normalization off, which makes keys specific to the machine's paths.

On a miss Kache runs nvcc with SOURCE_DATE_EPOCH=0, unless the build exports its own value, so __DATE__ and __TIME__ do not change between rebuilds. Set KACHE_NVCC_SOURCE_DATE_EPOCH=passthrough to stop Kache from setting it. The value is not part of the nvcc key, so a hit restores the object with the date it was first built with.

What passes through

Anything Kache does not model runs the real nvcc uncached, including:

• link and device-link (-dlink) modes, --lib, multi-source compiles;
• standalone -ptx / -cubin / -fatbin / --optix-ir emission, and preprocessing (-E);
• device debug (-G);
• machine-resolved values (-march=native, -arch=native, …). One key must never cover different objects, so these refuse even when allow-listed;
• dependency generation without an explicit output (-M, -MM, --generate-dependencies), the -MP and -MG modifiers, and -MD/-MMD without -MF. Combine -MD/-MMD with -MF <file> and leave out -MP and -MG;
• compiles without -o;
• -keep / --save-temps, profiling (--time), response files (@file);
• any flag outside the built-in table (see below). Common examples are -lineinfo, --use_fast_math, -Xcudafe, and --extended-lambda. Run the build with KACHE_LOG=kache=debug to see which flag blocked caching.

To cache a flag that Kache does not model but you know is safe, add it to the extra allow-list ([cc] extra_allowlist_flags or KACHE_CC_EXTRA_ALLOWLIST_FLAGS), which the nvcc adapter also honors. Flags that hide preprocessor inputs (-Xcompiler -I…, -Xcompiler -D…) stay refused: they would escape dependency tracking.

For nvcc, an allow-list entry matches the flag itself or the flag followed by =value, so --maxrregcount covers --maxrregcount=64. A value passed as a separate argument (--maxrregcount 64) counts as another unknown argument, so use the = form.

NVCC_PREPEND_FLAGS / NVCC_APPEND_FLAGS codegen tuning (-O2, -gencode …) is cached normally. Values that would change what is built or hide inputs (-G, -I…, -march=native) pass through uncached instead.