Apple Researchers Propose Weighted State Souping for Long-Context Retrieval

Apple's machine learning researchers, with co-authors from Harvard, published a paper describing weighted state souping, a method for getting state space models to handle long, crowded contexts. Building on prior state souping, where an SSM encodes chunks such as documents, tool definitions or memory records that can be cached and recombined, the new approach learns query-dependent weights for composing those cached states. The authors report gains on single- and multi-hop document question answering, needle-in-a-haystack retrieval, variable tracking and tool use, and describe a sparse variant with near-constant encoding cost as context length grows. The work was accepted to the PALM workshop at NeurIPS 2026; no code or product release is mentioned.