Test AI Research Tools on a Question With Changing Evidence
A polished answer can rely on an old revision or hide a disagreement. Use a small benchmark with a date cutoff and two named primary sources so each candidate faces the same task.
Worked example
Source freshness scoring sheet
Question: Which routes were suspended when the fictional Harbor Line timetable changed? Answer key: dated timetable A and its official correction B. Score: link opens / correct version / claim matches / correction acknowledged. One point each.
Steps
- Pick a research question specific enough to verify from primary documents. Save the document titles and publication dates in a private answer key before using any tool.
- Give each assistant the identical question, date cutoff, and requirement to label evidence by source. Save links and notes at the time you run it.
- Open each cited document. Mark whether it was available by the cutoff, whether the quoted point is present, and whether the answer presented any conflicting evidence.
What to check
- Read the original pages. A citation to a search result or a repost should not get the same score as the primary record.
- Keep timestamps. A tool's performance can change when its index updates, so compare reruns only with the same cutoff and answer key.