Apple Researchers Explain How Gating Holds Back State Space Models

Apple's machine learning team published a NeurIPS paper on why State Space Models, which offer constant memory and linear compute compared with Transformers, still trail on tasks that need in-context learning and exact retrieval. Through theory and experiments, the authors trace both the successes and shortcomings to the gating mechanism found in modern recurrent networks: gating pushes these models to first learn an in-weights memorization solution, which delays or even blocks convergence to a proper in-context learning solution, even where the architecture and memory capacity set no fundamental limit. Gating, though, often improves generalization to longer sequences. The work is by William L. Tong, Aryo Lotfi, Emmanuel Abbe and colleagues, with part of it done while at Apple, and the paper is available as a publication.