Apple researchers derive theoretical limits of language model alignment

Apple's machine learning team published a paper characterizing the information-theoretic limits of aligning language models under a KL-divergence budget. The authors give a closed-form expression for the largest expected reward gain achievable at a fixed KL cost, showing it depends on a Jeffreys divergence term rather than the square-root KL used in earlier analyses. They rewrite that expression as a covariance under the base model, which yields an estimator that predicts attainable alignment gains from base model samples alone. Experiments comparing the KL-reward Pareto frontier on safety and summarization tasks show best-of-N tracks the theoretical limit while PPO and GRPO fall short. The work was accepted at NeurIPS.