source: arxiv machine learning: operator-aware mixed-precision tolerance calibration for tensor kernels
level: technical
tensor kernel correctness tests usually rely on fixed absolute and relative tolerances chosen by hand. these thresholds are rarely updated and get copied across many projects. the authors collected element-wise error data from 26 gpuemu corpus entries across two data types, totaling 8,076 result rows from cloud gpu runs. they asked what absolute tolerance a correct kernel implementation would naturally justify. the answer was much tighter than the current hand-picked values.
the biggest tightening was for the attention_triton fp16 kernel, where the calibrated tolerance was 2,184 times smaller. the study focused on seven llm-style buggy kernel variants that had paired correct versions in the corpus. using the new per-operator and per-dtype tolerances, bug detection recall rose from 73.2% to 82.4%. that is an absolute gain of 9.3 percentage points, meaning 229 additional bugs were caught that the old tolerances missed.
the approach replaces guesswork with data-driven tolerance calibration. by observing how a correct kernel behaves across many runs, the method sets thresholds that are strict enough to flag real bugs without raising false positives. this makes automated testing more reliable for tensor operations commonly used in machine learning. the technique can be applied to any kernel corpus where correct reference outputs are available, helping developers catch subtle numerical errors earlier.
why it matters: data-driven tolerance calibration catches more numerical bugs in tensor kernels, improving reliability of ai and data science software.
source: arxiv machine learning: operator-aware mixed-precision tolerance calibration for tensor kernels