level: research
verifying transformer attention requires bounding the softmax function when pre-softmax scores lie in intervals. existing verifiers relax softmax independently of the downstream objective, which introduces unnecessary slack. this work shows that the exact optimum of this score-box problem is always at a vertex of the constraint box. a threshold structure theorem proves that after sorting the objective coefficients, the optimum is among only linearly many candidates. this leads to the vertex-softmax primitive with log-linear complexity in sequence length.
the paper proves that vertex-softmax is the tightest sound bound obtainable from score intervals alone. it also characterizes what additional structure, such as score correlations or score-value coupling, is needed for further improvement. the method integrates into crown-style convex relaxation frameworks for worst-case neuron analysis. experiments on vision transformers show that vertex-softmax reduces certified error by up to 22% compared to prior softmax relaxations, with runtime overhead under 5%.
the approach works by enumerating candidate vertices defined by the sorted objective coefficients. each candidate corresponds to a threshold where scores above it are set to the upper interval bound and those below to the lower bound. the exact softmax bound is then the maximum over these candidates. this avoids the looseness of interval bound propagation or linear relaxation methods that treat softmax as a black box. the result is a tighter, yet efficient, certification primitive for attention layers.
why it matters: tighter softmax bounds directly improve the certified robustness of transformer models, making safety guarantees more practical for real-world ai systems.