Evaluate an AI Research Tool on Scientific Methods Extraction

Scientific reading usually needs precise methods, not just a summary of the result. Build an answer key from an open methods section and compare each assistant against it.

Worked example

Fictional plant growth methods block

Group A received 12 hours of light and Group B received 8 hours. Seedlings shorter than 2 cm on day 1 were excluded. The primary outcome was mean leaf count on day 14. The duration of watering was not reported.
Fields: group conditions, exclusion, endpoint, watering duration.

Steps

  1. Choose a public methods section with named groups, a clear endpoint, and an exclusion. Independently write down its exact passages as the answer key.
  2. Ask every candidate to fill the same field table with the phrase and location. Require a missing-data marker for anything absent.
  3. Score each field for value, source location, and appropriate uncertainty. Recheck difficult fields with a second reader before drawing conclusions.

What to check

  • The last field should be marked absent. Check that day 14 stays attached to leaf count.
  • Track extraction accuracy separately from article discovery. A good search result does not establish that a tool read the full methods.

Find research tools to evaluate

References