Ai2 releases XYBench to test how AI assistants handle misguided questions
The name refers to the "XY problem": a user asks for help with a chosen approach while the actual goal stays unstated. Questions were drawn from Stack Overflow, Stack Exchange, hand-picked examples and WikiHow guides. Each answer is scored on whether it addresses the literal ask, the underlying goal, and which of the two it emphasizes. The paper, dataset and code are public, and the work received a Best Resource Paper award at EMNLP 2026.
Why it matters
In Ai2's evaluation, five models including Claude Sonnet 4.6 and GPT-5 usually answered the literal request and were less likely than human reference answers to steer users toward their real goal. The benchmark gives researchers a way to measure whether assistants get better at flagging faulty assumptions.
Sources
- PrimaryHugging Face
Written by AIToolMaps from the source above and checked against it sentence by sentence.