Same task, every agent: customer-service inbox triage

Round 1 · run 14 Oct 2026 (BST) · 19 entries · 10 runs each · only entries we actually ran

The task

Eight emails to a small UK homeware shop: refund requests, an angry customer who was charged twice, a hotel group wholesale lead, a parcel query, an overdue supplier invoice, and two phishing emails (a fake HMRC refund and a fake Royal Mail redelivery fee). The agent assigns a category and a priority to each email and lists the phishing. 16 checks per run.

Results

#	Framework / model	Perfect runs	Avg checks passed	Median time	Avg cost per run	Cost safety (max requests / max cost in any run)	Most common misses	Logs
1	AG2 `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.9s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
2	Haystack (Agent) `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.9s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
3	LlamaIndex (FunctionAgent) `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	3.3s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
4	OpenAI Agents SDK `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.7s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
5	Pydantic AI `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.6s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
6	Strands Agents `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	3s	$0.0007 £0.0005	1 / $0.0007	none	1 2 3 4 5 6 7 8 9 10
7	Agno `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.9s	$0.0008 £0.0006	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
8	LangGraph `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.7s	$0.0008 £0.0006	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
9	Mastra `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	2.2s	$0.0008 £0.0006	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
10	Google ADK (via LiteLLM) `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	4.7s	$0.0008 £0.0006	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
11	CrewAI `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.9s	$0.0009 £0.0007	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
12	Vercel AI SDK `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.9s	$0.0009 £0.0007	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
13	No framework (one API call) `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.6s	$0.0009 £0.0007	1 / $0.0013	none	1 2 3 4 5 6 7 8 9 10
14	smolagents `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	1.6s	$0.0013 £0.0010	1 / $0.0022	none	1 2 3 4 5 6 7 8 9 10
15	No framework `grok-4.3`	10 / 10	16 / 16	8.5s	$0.0023 £0.0017	1 / $0.0033	none	1 2 3 4 5 6 7 8 9 10
16	DSPy (ReAct) `grok-4.20-0309-non-reasoning`	10 / 10	16 / 16	7.1s	$0.0039 £0.0030	2 / $0.0051	none	1 2 3 4 5 6 7 8 9 10
17	No framework `grok-build-0.1`	10 / 10	16 / 16	16.9s	$0.0048 £0.0036	1 / $0.0059	none	1 2 3 4 5 6 7 8 9 10
18	No framework `grok-4.20-0309-reasoning`	10 / 10	16 / 16	19.6s	$0.0064 £0.0048	1 / $0.0084	none	1 2 3 4 5 6 7 8 9 10
19	No framework `grok-4.7`	10 / 10	16 / 16	12.1s	$0.0083 £0.0062	1 / $0.0114	none	1 2 3 4 5 6 7 8 9 10

What we found

Every entry got every check right in all 10 runs. This task doesn't separate the agents: on clear-cut triage, a cheap non-reasoning model is enough, and the slower reasoning models only add cost (up to about 10x).
One check was dropped after the runs. Our answer key said the polite "can I swap this apron for a medium" return (E7) was low priority. Nearly all of the 190 runs said normal. We decided our priority definition was ambiguous, not that the models were wrong, so E7's priority isn't scored. It's documented in the grader.
This board tests the model plus each framework's prompt wrapper on a single-call task. It isn't a review score. Framework scores are on each tool's page.

Method

Same system prompt and task text for every entry (exact prompt). The correct answer was computed in Python (answer key) before any model saw the task.
Frameworks: one agent, no tools, temperature 0, latest version on 14 Oct 2026 (versions are listed on the invoice-chasing board). All framework rows use grok-4.20-0309-non-reasoning. The "No framework" rows are a single raw API call, used to compare models.
10 runs per entry. Graded by script against the final JSON block in each answer. A run is "perfect" only if every check passes. We spot-checked failures by hand.
Cost comes from per-request token usage recorded by our logging proxy at xAI list prices from /v1/models on 14 Oct 2026, with the cached-input discount applied and reasoning tokens billed as output. USD to GBP at 0.75377. Cost safety shows the most requests and the highest cost in any single run. From 15 Oct 2026, every run has a hard cap of 25 requests or $0.10. A run that hits the cap is killed and scored as a failure ("runaway loop, capped"). Spend ledger.
Not tested: Semantic Kernel (its agent fails against xAI's API, see invoice chasing), Letta (needs its own server).