ResearchNov 17, 2026, 9:53 AM EST

Runta and Baseten benchmark 22 coding agent harnesses across five models

Runta published FrontierHarness Eval v1.1, a harness leaderboard built with Baseten, expanding its earlier single-model version to 22 harness configurations run against five models: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse and Kimi-K3. The main finding is that harness rankings shift with the model: Codex CLI was cheapest per task with GLM-5.3 but 16th with Kimi-K3, and with a fixed model the priciest harness cost up to five times the cheapest one. Step count correlated with cost per pass; prompt size, tool count and cache hit rate did not.

Why it matters

If you change which model powers your coding agent, the harness you pair it with may need to change too: the same harness can move from cheapest to near the bottom of the cost ranking depending on the model, without a matching gain in pass rate.

Sources

Written by AIToolMaps from the source above and checked against it sentence by sentence.

Comments

No comments yet