Same task, every agent: messy supplier invoice extraction
Round 1 · run 14 Oct 2026 (BST) · 19 entries · 10 runs each · only entries we actually ran
The task
A forwarded supplier invoice email from a UK timber merchant. The agent has to pull out the supplier, invoice number, dates (written DD/MM/YYYY, due 30 days after the invoice date), VAT number, and net, VAT and gross totals. The traps: a 5% trade discount that applies to two lines only, a zero-rated book, and a figure from a previous invoice in the footer. Correct totals: net £430.80, VAT £83.76, gross £514.56, due 2026-10-03.
Results
| # | Framework / model | Perfect runs | Avg checks passed | Median time | Avg cost per run | Cost safety (max requests / max cost in any run) | Most common misses | Logs |
|---|---|---|---|---|---|---|---|---|
| 1 | No frameworkgrok-4.3 | 10 / 10 | 8 / 8 | 7.9s | $0.0022 £0.0017 | 1 / $0.0025 | none | 1 2 3 4 5 6 7 8 9 10 |
| 2 | DSPy (ReAct)grok-4.20-0309-non-reasoning | 10 / 10 | 8 / 8 | 8.7s | $0.0047 £0.0035 | 2 / $0.0061 | none | 1 2 3 4 5 6 7 8 9 10 |
| 3 | No frameworkgrok-4.20-0309-reasoning | 10 / 10 | 8 / 8 | 13.9s | $0.0047 £0.0036 | 1 / $0.0058 | none | 1 2 3 4 5 6 7 8 9 10 |
| 4 | No frameworkgrok-build-0.1 | 10 / 10 | 8 / 8 | 18.2s | $0.0053 £0.0040 | 1 / $0.0067 | none | 1 2 3 4 5 6 7 8 9 10 |
| 5 | No frameworkgrok-4.7 | 10 / 10 | 8 / 8 | 16.7s | $0.0118 £0.0089 | 1 / $0.0139 | none | 1 2 3 4 5 6 7 8 9 10 |
| 6 | Strands Agentsgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 2.6s | $0.0004 £0.0003 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 7 | Haystack (Agent)grok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.5s | $0.0004 £0.0003 | 1 / $0.0004 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 8 | LangGraphgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.2s | $0.0004 £0.0003 | 1 / $0.0004 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 9 | LlamaIndex (FunctionAgent)grok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 3s | $0.0004 £0.0003 | 1 / $0.0004 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 10 | OpenAI Agents SDKgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.3s | $0.0004 £0.0003 | 1 / $0.0004 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 11 | Pydantic AIgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.3s | $0.0004 £0.0003 | 1 / $0.0004 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 12 | AG2grok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.5s | $0.0005 £0.0004 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 13 | Mastragrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.8s | $0.0005 £0.0004 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 14 | Agnogrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.7s | $0.0005 £0.0004 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 15 | Vercel AI SDKgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.6s | $0.0005 £0.0004 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 16 | No framework (one API call)grok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.2s | $0.0005 £0.0004 | 2 / $0.0013 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 17 | Google ADK (via LiteLLM)grok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 4.4s | $0.0006 £0.0004 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 18 | CrewAIgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.7s | $0.0006 £0.0005 | 1 / $0.0009 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
| 19 | smolagentsgrok-4.20-0309-non-reasoning | 0 / 10 | 5 / 8 | 1.9s | $0.0010 £0.0008 | 1 / $0.0019 | gross total (10/10); net total (10/10); vat total (10/10) | 1 2 3 4 5 6 7 8 9 10 |
What we found
- Every framework on the non-reasoning model got the totals wrong in all 10 runs. The text fields were always right. The usual error was a net of £480.33, from applying the discount to the wrong lines. The framework made no difference: every one matched the no-framework baseline.
- DSPy (ReAct) was right 10/10 on the same model. Its module makes the model write out its reasoning before answering, which costs about 10x the tokens of the plain frameworks (still under half a US cent).
- Every reasoning-capable model we tried (grok-4.20 reasoning, grok-4.3, grok-4.7, grok-build-0.1) was right 10/10. grok-4.3 was the cheapest at about $0.002 per run.
- This board tests the model plus each framework's prompt wrapper on a single-call task. It isn't a review score. Framework scores are on each tool's page.
Method
- Same system prompt and task text for every entry (exact prompt). The correct answer was computed in Python (answer key) before any model saw the task.
- Frameworks: one agent, no tools, temperature 0, latest version on 14 Oct 2026 (versions are listed on the invoice-chasing board). All framework rows use
grok-4.20-0309-non-reasoning. The "No framework" rows are a single raw API call, used to compare models. - 10 runs per entry. Graded by script against the final JSON block in each answer. A run is "perfect" only if every check passes. We spot-checked failures by hand.
- Cost comes from per-request token usage recorded by our logging proxy at xAI list prices from
/v1/modelson 14 Oct 2026, with the cached-input discount applied and reasoning tokens billed as output. USD to GBP at 0.75377. Cost safety shows the most requests and the highest cost in any single run. From 15 Oct 2026, every run has a hard cap of 25 requests or $0.10. A run that hits the cap is killed and scored as a failure ("runaway loop, capped"). Spend ledger. - Not tested: Semantic Kernel (its agent fails against xAI's API, see invoice chasing), Letta (needs its own server).