Bake-off on your invoices
Accuracy on synthetic invoices proves the software works, not that it reads your invoices. A bake-off measures it on a sample of your own invoices, scored against what your AP team actually posted.
What the agency provides
Section titled “What the agency provides”- Invoices: 500 or more recent originals as received: PDFs, scans, phone photos, EDI or XML files and email attachments. Include the top suppliers by volume, utilities, multi-page invoices and credit memos. Do not clean them up.
- The AP export for the same invoices: one CSV from the ERP with a row per invoice (number, supplier, date, amount, currency, type, PO number, tax, freight). Oracle EBS, Workday and Tyler Munis exports work as they come.
- How files and rows connect: a column with each row’s file name, or file names that contain the invoice number.
Nothing leaves the agency’s environment. The kit runs on a laptop or VM the agency controls; only model calls go out.
Run it
Section titled “Run it”export ANTHROPIC_API_KEY=...pnpm bakeoff --dir /data/invoices --export /data/ap-export.csv --sample 500The sample is stratified: top suppliers in proportion, with credit memos and non-PO invoices at their real share. The same seed always picks the same files.
Read the results
Section titled “Read the results”| Measure | Why it matters |
|---|---|
| Field accuracy | How often each value is right |
| Straight-through rate | How many invoices need no one |
| Straight-through precision | Of those, how many were exactly right |
| Errors that would post without review | The number that matters most: should be zero |
| Accuracy after review | What would reach the ERP |
report.md is the summary to share; results.csv has a row per invoice so AP can check any disagreement in minutes.