Evaluate an AI Research Tool on Scientific Methods Extraction
Scientific reading usually needs precise methods, not just a summary of the result. Build an answer key from an open methods section and compare each assistant against it.
Worked example
Fictional plant growth methods block
Group A received 12 hours of light and Group B received 8 hours. Seedlings shorter than 2 cm on day 1 were excluded. The primary outcome was mean leaf count on day 14. The duration of watering was not reported. Fields: group conditions, exclusion, endpoint, watering duration.
Steps
- Choose a public methods section with named groups, a clear endpoint, and an exclusion. Independently write down its exact passages as the answer key.
- Ask every candidate to fill the same field table with the phrase and location. Require a missing-data marker for anything absent.
- Score each field for value, source location, and appropriate uncertainty. Recheck difficult fields with a second reader before drawing conclusions.
What to check
- The last field should be marked absent. Check that day 14 stays attached to leaf count.
- Track extraction accuracy separately from article discovery. A good search result does not establish that a tool read the full methods.