Compare PDF Chat Tools With a Page-Citation Test

Use a two-page file you create yourself so the answer key is unambiguous. The same file tests three separate abilities: retrieving a table value, finding a qualifying note, and acknowledging an absent fact.

Worked example

A two-page PDF test fixture

Page 1: Route A, daily cap 12 rides. Route B, daily cap 20 rides.
Page 2: Route B's cap is 10 rides on Sundays.
Questions: What is the normal Route B cap? What is the Sunday cap? How much does a ticket cost?
Expected: 20 on page 1, 10 on page 2, and cost absent.

Steps

  1. Create a document with the fixture, insert a page break, and export to PDF. Verify the text is selectable and the page order matches the answer key.
  2. Upload the same file to each candidate. Ask the three questions one at a time and capture the quoted evidence, any link, and the reported page label.
  3. Open the citation where available. Score the values and the location separately, including whether the reader can distinguish the physical PDF page from a printed page number.

What to check

  • An answer that gives the correct number but points to the wrong page gets the value point only.
  • Test scanned PDF support separately with a controlled OCR version so OCR errors and citation errors can be compared.

Explore PDF chat tools

References