A stricter test finds most LLM-written GPU kernels are broken

A new paper argues that the correctness scores reported by systems that use language models to write GPU kernels are largely an illusion. Those systems usually test a generated kernel by running it on random inputs at one fixed shape and comparing against a reference within a numerical tolerance. The authors show a kernel can pass that check while returning the wrong data type, giving different answers on repeated runs, or failing entirely on a different input shape.

Their alternative is a contract-grade verifier: twelve adversarial gates, each a property a correct kernel must hold, several of them tolerance-free so they cannot be waved away by loosening a threshold. Run against 2,638 machine-generated kernels that existing systems had already accepted, the verifier found 39.5% broken beyond any tolerance argument and 62.1% carrying at least one violation. The disagreement is lopsided: the loose test accepted 1,487 kernels the strict one rejects, against only 14 the other way. The paper also ships a native Blackwell backward pass for gated-linear-recurrence models as a worked example of a kernel that clears every gate.

Why it matters

If you rely on LLM-generated CUDA kernels, or trust leaderboard numbers from kernel-writing agents, this says the acceptance bar is far too low and much of what passes would break in production. Before shipping a generated kernel, test it across input shapes and for run-to-run determinism, not just one shape within a tolerance.