Same task, every agent: weekly cash-flow summary from a bank CSV

Round 1 · run 14 Oct 2026 (BST) · 19 entries · 10 runs each · only entries we actually ran

The task

A bank export of 21 transactions (card takings, an Amazon payout, rent, wages, a VAT payment to HMRC, stock, utilities) with an opening balance. The agent groups them into weeks starting Monday, gives money in, money out, net and closing balance for each week, and names the biggest outflow category. Two Sunday transactions test the week boundaries. Correct final balance: £9,159.59. Biggest outflow: Payroll, £4,800.00. 21 checks per run.

Results

#	Framework / model	Perfect runs	Avg checks passed	Median time	Avg cost per run	Cost safety (max requests / max cost in any run)	Most common misses	Logs
1	No framework `grok-build-0.1`	10 / 10	21 / 21	34s	$0.0109 £0.0082	1 / $0.0132	none	1 2 3 4 5 6 7 8 9 10
2	No framework `grok-4.20-0309-reasoning`	10 / 10	21 / 21	45.9s	$0.0153 £0.0115	1 / $0.0179	none	1 2 3 4 5 6 7 8 9 10
3	No framework `grok-4.7`	10 / 10	21 / 21	32.2s	$0.0270 £0.0203	1 / $0.0288	none	1 2 3 4 5 6 7 8 9 10
4	DSPy (ReAct) `grok-4.20-0309-non-reasoning`	9 / 10	19.2 / 21	49.9s	$0.0229 £0.0173	3 / $0.0275	2026-09-07 closing balance (1/10); 2026-09-07 money in (1/10); 2026-09-07 net (1/10)	1 2 3 4 5 6 7 8 9 10
5	No framework `grok-4.3`	8 / 10	20.1 / 21	13.8s	$0.0042 £0.0031	1 / $0.0047	2026-09-07 closing balance (1/10); 2026-09-07 money in (1/10); 2026-09-07 net (1/10)	1 2 3 4 5 6 7 8 9 10
6	CrewAI `grok-4.20-0309-non-reasoning`	0 / 10	5.6 / 21	2.8s	$0.0011 £0.0008	1 / $0.0017	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
7	No framework (one API call) `grok-4.20-0309-non-reasoning`	0 / 10	5.2 / 21	2.8s	$0.0011 £0.0008	1 / $0.0012	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
8	Pydantic AI `grok-4.20-0309-non-reasoning`	0 / 10	4.7 / 21	2.7s	$0.0011 £0.0008	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
9	LlamaIndex (FunctionAgent) `grok-4.20-0309-non-reasoning`	0 / 10	4.4 / 21	4.5s	$0.0009 £0.0007	1 / $0.0010	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
10	Google ADK (via LiteLLM) `grok-4.20-0309-non-reasoning`	0 / 10	4.4 / 21	5.8s	$0.0011 £0.0009	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
11	Strands Agents `grok-4.20-0309-non-reasoning`	0 / 10	4.3 / 21	4s	$0.0010 £0.0008	1 / $0.0017	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
12	AG2 `grok-4.20-0309-non-reasoning`	0 / 10	4.3 / 21	3.1s	$0.0011 £0.0008	1 / $0.0012	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
13	Vercel AI SDK `grok-4.20-0309-non-reasoning`	0 / 10	4.3 / 21	3.1s	$0.0011 £0.0008	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
14	Mastra `grok-4.20-0309-non-reasoning`	0 / 10	4.2 / 21	3.2s	$0.0011 £0.0008	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
15	Haystack (Agent) `grok-4.20-0309-non-reasoning`	0 / 10	4.2 / 21	2.9s	$0.0011 £0.0009	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
16	LangGraph `grok-4.20-0309-non-reasoning`	0 / 10	4.1 / 21	2.9s	$0.0011 £0.0008	1 / $0.0012	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
17	OpenAI Agents SDK `grok-4.20-0309-non-reasoning`	0 / 10	4.1 / 21	2.9s	$0.0012 £0.0009	1 / $0.0016	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
18	Agno `grok-4.20-0309-non-reasoning`	0 / 10	3.8 / 21	3.2s	$0.0011 £0.0008	1 / $0.0012	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10
19	smolagents `grok-4.20-0309-non-reasoning`	0 / 10	3.8 / 21	2.8s	$0.0013 £0.0010	1 / $0.0026	2026-09-07 closing balance (10/10); 2026-09-07 money in (10/10); 2026-09-07 money out (10/10)	1 2 3 4 5 6 7 8 9 10

What we found

Every framework on the non-reasoning model failed every run, averaging 4 to 6 checks out of 21. The weekly sums were wrong, usually from the first week on. The category question was usually right.
DSPy (ReAct) on the same model: 9/10 perfect, but it took about 50 seconds and 20x the cost of the other frameworks.
The reasoning models grok-4.20 reasoning, grok-build-0.1 and grok-4.7 were perfect 10/10. grok-4.3 was perfect 8/10 and the cheapest of them (about $0.004 per run).
Practical advice: don't ask a non-reasoning model to add up a bank statement in its head. Use a reasoning model or give the agent real code. Our first try at giving agents a calculator tool went badly; see the pilot below.
This board tests the model plus each framework's prompt wrapper on a single-call task. It isn't a review score. Framework scores are on each tool's page.

Pilot: the same task with a calculator tool (not ranked)

We gave six frameworks a calculate tool (source) and told them to use it for every sum. 1 run each, non-reasoning model. This is a pilot, not a leaderboard, because one run each proves nothing about ranking. We're publishing it for the cost-safety lesson.

Framework	Outcome	Model requests	Cost	Cost safety
Agno	No answer. We killed it after about 7 minutes. The prompt grew to 41,657 tokens per request.	327	$1.59	Runaway loop. We had set `tool_call_limit=60` and it still made 327 requests. log
Pydantic AI	Crashed (error raised after the tool loop)	50	$0.042	Stopped on its own log
LangGraph	"Sorry, need more steps": hit our recursion limit of 80 after 39 calculator calls	40	$0.033	Stopped at its limit log
OpenAI Agents SDK	Hit our `max_turns=40`	40	$0.029	Stopped at its limit log
Strands Agents	Connection error mid-stream. This may be our proxy's fault rather than Strands', so don't read it as a Strands failure	78	$0.10	Unclear log
smolagents	Answered, 4/21 checks	11	$0.018	OK log

Lesson: one calculator call per addition turns a 1-request task into 40 to 300+ requests. Set your own hard cap on requests and spend, and don't rely on the framework's limit. We now cap every test run at 25 requests or $0.10.

Method

Same system prompt and task text for every entry (exact prompt). The correct answer was computed in Python (answer key) before any model saw the task.
Frameworks: one agent, no tools, temperature 0, latest version on 14 Oct 2026 (versions are listed on the invoice-chasing board). All framework rows use grok-4.20-0309-non-reasoning. The "No framework" rows are a single raw API call, used to compare models.
10 runs per entry. Graded by script against the final JSON block in each answer. A run is "perfect" only if every check passes. We spot-checked failures by hand.
Cost comes from per-request token usage recorded by our logging proxy at xAI list prices from /v1/models on 14 Oct 2026, with the cached-input discount applied and reasoning tokens billed as output. USD to GBP at 0.75377. Cost safety shows the most requests and the highest cost in any single run. From 15 Oct 2026, every run has a hard cap of 25 requests or $0.10. A run that hits the cap is killed and scored as a failure ("runaway loop, capped"). Spend ledger.
Not tested: Semantic Kernel (its agent fails against xAI's API, see invoice chasing), Letta (needs its own server).