Compare PDF Chat Tools With a Page-Citation Test
Use a two-page file you create yourself so the answer key is unambiguous. The same file tests three separate abilities: retrieving a table value, finding a qualifying note, and acknowledging an absent fact.
Worked example
A two-page PDF test fixture
Page 1: Route A, daily cap 12 rides. Route B, daily cap 20 rides. Page 2: Route B's cap is 10 rides on Sundays. Questions: What is the normal Route B cap? What is the Sunday cap? How much does a ticket cost? Expected: 20 on page 1, 10 on page 2, and cost absent.
Steps
- Create a document with the fixture, insert a page break, and export to PDF. Verify the text is selectable and the page order matches the answer key.
- Upload the same file to each candidate. Ask the three questions one at a time and capture the quoted evidence, any link, and the reported page label.
- Open the citation where available. Score the values and the location separately, including whether the reader can distinguish the physical PDF page from a printed page number.
What to check
- An answer that gives the correct number but points to the wrong page gets the value point only.
- Test scanned PDF support separately with a controlled OCR version so OCR errors and citation errors can be compared.