level: research
linear attention models compress the key-value cache into a fixed-size recurrent state, which avoids the growing memory cost of standard softmax attention. however, this compression can hurt exact state tracking and long-context recall. the paper proposes semidirect fourier delta attention (sfda), which builds on kimi delta attention but replaces real diagonal decay with block-rotational fourier control. the state update uses a complex diagonal matrix with learnable phase angles, allowing the model to maintain cyclic or oscillatory memory patterns.
the main technical contribution is a constructive chunk-wy factorization for the product of state transition matrices. this factorization expresses the product over a chunk as a low-rank update to a diagonal matrix, with rank growth bounded inside fixed-size chunks. it yields an exact affine chunk transfer, formal stability and complexity bounds, and a compact description of memory as phase-plus-low-rank. the authors verify the algebra numerically and test on toy state-tracking tasks.
in experiments, sfda learns cyclic memory where a phase-disabled baseline fails. the phase control lets the model track periodic patterns without extra parameters. the chunk-wise formulation also makes training and inference efficient by limiting the rank of the low-rank component. while the paper focuses on theory and simple tasks, the approach could extend to larger models needing better long-range memory with fixed state size.
why it matters: this method could improve long-context performance in efficient transformers by adding phase-controlled memory, useful for tasks like document understanding or time-series modeling where cyclic patterns matter.