Test AI Research Tools on a Question With Changing Evidence

A polished answer can rely on an old revision or hide a disagreement. Use a small benchmark with a date cutoff and two named primary sources so each candidate faces the same task.

Worked example

Source freshness scoring sheet

Question: Which routes were suspended when the fictional Harbor Line timetable changed?
Answer key: dated timetable A and its official correction B.
Score: link opens / correct version / claim matches / correction acknowledged. One point each.

Steps

  1. Pick a research question specific enough to verify from primary documents. Save the document titles and publication dates in a private answer key before using any tool.
  2. Give each assistant the identical question, date cutoff, and requirement to label evidence by source. Save links and notes at the time you run it.
  3. Open each cited document. Mark whether it was available by the cutoff, whether the quoted point is present, and whether the answer presented any conflicting evidence.

What to check

  • Read the original pages. A citation to a search result or a repost should not get the same score as the primary record.
  • Keep timestamps. A tool's performance can change when its index updates, so compare reruns only with the same cutoff and answer key.

Find research assistants to test

References