Apple researchers train small networks to speed up RAG retrieval

Apple researchers have published a method that trains small neural networks to route search queries toward the most promising cluster of documents, aiming to lower the cost of retrieval for recommendation, search and retrieval-augmented generation. In a paper presented at ICML 2026, the team treats the best-match score over a set of candidate vectors as a support function and learns to approximate it with input convex neural networks called SupportNets, alongside a companion KeyNet that predicts the matching vector directly. Training happens once, offline, for a fixed database, using queries sampled from the distribution a system expects to see. Experiments used Natural Questions with roughly 2.6 million passages split into 128 clusters, testing models sized from 1% to 20% of the database.