level: research
subquadratic attention alternatives often struggle with multi-dimensional data like images or 3d volumes. standard convolutions cannot capture long-range dependencies and lack input sensitivity. recurrent models force data into a one-dimensional scan order, breaking natural spatial structure. hyenand solves this by operating directly on the native geometry of multi-dimensional data. it uses implicitly parametrized global convolutional kernels that adapt to the input, providing a global receptive field without quadratic cost.
the method relies on long convolutions with input-dependent kernels, achieving o(l log l) complexity. a custom cuda implementation called nsubq fuses the fft-based convolution path, turning theoretical scaling into real speed gains. experiments cover long-context genomics, computer vision, medical imaging, and pde modeling. pure hyenand stacks match the accuracy of strong attention baselines while running faster. hybrid configurations that combine hyenand with attention are also explored.
by avoiding rasterization, hyenand preserves spatial relationships that are critical for tasks like segmentation or physical simulation. the operator is global and input-dependent, meaning it can model complex dependencies across entire images or volumes without the memory blowup of attention. the work shows that subquadratic methods can be practical for high-dimensional data, not just text. the nsubq kernel makes the approach viable for real-world workloads where both accuracy and speed matter.
why it matters: it enables efficient processing of images, 3d volumes, and scientific data without losing the global context that attention provides, making large-scale vision and simulation models more practical.