K2-Horizon Releases Six Fully Open Language Models, Including a 375B MoE
K2-Horizon is a family of six openly released language models, ranging from a 0.9B dense model to a 375B-A23B mixture-of-experts model, built for reasoning and agentic work in math, science, software engineering, deep research and office automation. The authors say the models lead or are competitive with strong open models of their size on agentic and coding benchmarks. Two attention changes, Mixture-of-Value Attention and softplus attention gating, let a 36B-A4B model nearly match a 32B dense model trained on the same data with one-seventh the active parameters. A companion release, Uno, provides diffusion adapters that speed decoding by up to 2.2x without a draft model. Checkpoints from every training stage, pre-training data, logs, and training and evaluation code are published, along with the data-building pipelines.
Why it matters
The release discloses the data pipelines and intermediate checkpoints that open model releases usually withhold, so teams can audit or reproduce the training instead of only using the weights.
Sources
- PrimaryHugging Face Papers
Written by AIToolMaps from the source above and checked against it sentence by sentence.