Same task, every agent: invoice chasing
Rounds 1 and 2 · run 14 Oct 2026 · model grok-4.20-0309-non-reasoning (xAI) · 17 frameworks · only entries we actually ran
The task
Five unpaid invoices for a UK small business (CSV), today = 14 Oct 2026. The agent must (1) compute 20% VAT and gross for each invoice, (2) list the overdue ones, (3) total the gross overdue, and (4) draft a polite UK-English chaser to the customer with the oldest overdue invoice. Correct answer (computed in Python): INV-101, INV-102 and INV-105 overdue, £3,150.60, chaser to Tyne Accountants for £1,170.60.
Results
| # | Framework | Fully correct runs | Requested checks passed | Wall time per run (s) | Avg cost per run | Notes and logs |
|---|---|---|---|---|---|---|
| 1 | CrewAI crewai 1.15.28 | 3 / 3 | 12 / 12 | 3.7 / 3.7 / 3.7 | $0.0015 £0.0011 | Own agent prompt template. Correct every run. Logs: 1 2 3 |
| 2 | smolagents smolagents 1.26.0 (ToolCallingAgent, no tools) | 3 / 3 | 12 / 12 | 2.7 / 2.8 / 3.2 | $0.0024 £0.0018 | Answer comes back via its final_answer tool. Terse output, no extra figures. Logs: 1 2 3 |
| 3 | No framework, reasoning model one Chat Completions call, grok-4.20-0309-reasoning | 3 / 3 | 12 / 12 | 9.3 / 9.9 / 10.1 | $0.0039 £0.0030 | Baseline using the reasoning variant of the same model. Correct every run, with no wrong extra figures. About 1,000 reasoning tokens per run, billed as output. Logs: 1 2 3 |
| 4 | DSPy dspy 3.4.1 (ReAct, no tools) | 3 / 3 | 12 / 12 | 8.9 / 6.9 / 9.0 | $0.0043 £0.0032 | Correct every run. Our regex grader missed its one-line overdue list, so we checked it by hand. DSPy ReAct asks the model for a reasoning step before it answers, which uses about 3x the tokens. Logs: 1 2 3 |
| 5 | AG2 ag2 1.1.2 (ag2.Agent, OpenAIConfig) | 1 / 3 | 12 / 12 | 3.7 / 3.7 / 3.3 | $0.0014 £0.0010 | Runs 2 and 3 said INV-101 is 26 days overdue (it is 25). Logs: 1 2 3 |
| 6 | LangGraph langgraph 1.2.15 (create_react_agent) | 1 / 3 | 11 / 12 | 3.2 / 3.4 / 3.2 | $0.0018 £0.0013 | Run 1: chaser went to the wrong customer (INV-101), with "57 days overdue". Run 2: "26 days" (actually 25). Logs: 1 2 3 |
| 7 | Vercel AI SDK ai 7.0.140 + @ai-sdk/openai-compatible 3.0.68 (Node 20) | 0 / 3 | 12 / 12 | 4.0 / 3.6 / 3.8 | $0.0014 £0.0010 | Every run said 26 days. Logs: 1 2 3 |
| 8 | LlamaIndex llama-index-core 0.14.25 (FunctionAgent, OpenAILike) | 0 / 3 | 12 / 12 | 4.1 / 4.3 / 4.0 | $0.0014 £0.0010 | Every run said 26 days. Logs: 1 2 3 |
| 9 | Haystack haystack-ai 3.3.0 (Agent, OpenAIChatGenerator) | 0 / 3 | 12 / 12 | 5.2 / 3.8 / 3.5 | $0.0014 £0.0010 | Every run said 26 days. Logs: 1 2 3 |
| 10 | No framework, non-reasoning model one Chat Completions call, grok-4.20-0309-non-reasoning | 0 / 3 | 12 / 12 | 3.9 / 3.7 / 4.0 | $0.0014 £0.0010 | Baseline: the same prompt with no framework. Every run said 26 days. Logs: 1 2 3 |
| 11 | Mastra @mastra/core 1.77.0 (Node 20) | 0 / 3 | 12 / 12 | 3.6 / 3.8 / 3.5 | $0.0014 £0.0011 | Every run said 26 days. Logs: 1 2 3 |
| 12 | Strands Agents strands-agents 1.61.0 (OpenAIModel) | 0 / 3 | 12 / 12 | 3.9 / 4.3 / 3.8 | $0.0015 £0.0011 | Days overdue for INV-101 given as 26, 37 and 26 (it is 25). Logs: 1 2 3 |
| 13 | Pydantic AI pydantic-ai 2.56.0 | 0 / 3 | 12 / 12 | 3.2 / 3.7 / 2.9 | $0.0018 £0.0013 | Every run said INV-101 is "26 days overdue" (it is 25). Logs: 1 2 3 |
| 14 | OpenAI Agents SDK openai-agents 0.23.1 (Chat Completions model pointed at xAI) | 0 / 3 | 12 / 12 | 3.4 / 3.3 / 3.0 | $0.0018 £0.0013 | INV-101 days overdue given as 37, 26, 26 (it is 25). Logs: 1 2 3 |
| 15 | Agno agno 3.1.3 (OpenAILike) | 0 / 3 | 12 / 12 | 3.4 / 3.4 / 3.4 | $0.0018 £0.0013 | Every run said INV-101 is "26 days overdue" (it is 25). Logs: 1 2 3 |
| 16 | Google ADK google-adk 2.12.0 (LlmAgent via LiteLlm, OpenAI-compatible) | 0 / 3 | 11 / 12 | 7.6 / 6.3 / 6.7 | $0.0015 £0.0011 | Run 1 sent the chaser to the wrong customer (INV-101, "56 days overdue"). Runs 2 and 3 said 35 and 26 days. Logs: 1 2 3 |
| 17 | Semantic Kernel semantic-kernel 1.45.0 (ChatCompletionAgent) | 0 / 2 | 0 / 8 | n/a | $0.0000 £0.0000 | Did not run. xAI rejected every request with HTTP 400: "Only message of role user can have a name". The agent puts its name on non-user messages. We did not patch the framework. 2 attempts, both logged. Logs: 1 2 |
"Fully correct" means all 4 requested checks passed and the output contained no wrong extra figures, such as an incorrect "days overdue". A wrong number in a chaser email is still wrong. Costs come from each framework's reported token usage at xAI's list price ($1.25 per 1M input, $2.50 per 1M output, checked 14 Oct 2026 on docs.x.ai). Cached-token discounts are applied only where the framework reported them (CrewAI). Elsewhere we use the full input price, so those costs are upper bounds. 1 USD = £0.75377 (ECB rate for 14 Oct 2026 via frankfurter.app). Cost sheet.
How to read this (important)
- Round 2 confirms it. A plain API call with no framework gave the same answers as LlamaIndex, Haystack, the Vercel AI SDK, Mastra and most of the others, down to the same wrong “26 days”. The entries that were right every run were the reasoning model and DSPy, whose ReAct module makes the model write out its reasoning before it answers. On this task, thinking time mattered more than the choice of framework.
- Round 1 mostly measures the model and the prompt wrapper, not the framework. The task needs no tools, so each framework makes a single model call. LangGraph, Pydantic AI, the OpenAI Agents SDK and Agno pass our prompt through almost unchanged and gave near-identical answers, down to the same off-by-one day count. CrewAI and smolagents wrap the prompt in their own templates. Here that happened to produce outputs with no wrong extra figures, but that isn't evidence they're better frameworks.
- 3 runs per framework at temperature 0 is a consistency check. It doesn't give statistical confidence.
- This leaderboard isn't a review score. The scored reviews are separate: see CrewAI 86/100.
Methodology
- The same system text (role, goal and "UK standard VAT rate is 20%") and task text for every framework (shared prompt). CrewAI uses its original test script, where the same text sits in role/goal/backstory.
- One agent, no tools, the same model, temperature 0. The latest PyPI version of each framework on 14 Oct 2026, each in its own Python 3.12 virtualenv on a Linux box. Scripts: smolagents, LangGraph, Pydantic AI, OpenAI Agents SDK, Agno.
- 3 runs each. Graded by a script and then checked by hand. The hand check caught errors the script missed (days overdue) and a grader bug (it read "not overdue" as overdue).
- Every run is published, including a crashed run caused by a bug in our own script, which we re-ran.
- Round 2 entries: requests went through a small logging proxy on our box. It adds the API key, records each request's token usage, and asks for usage on streamed replies. It also drops
frequency_penalty/presence_penaltywhen they are 0, because xAI rejects them on some models (see build log #2). Costs come from those records, with the cached-token discount included (ledger, raw usage). Scripts: shared prompt loader, plussrc-leaderboard_run_*.py.txtfor each Python framework and the no-framework baseline, Vercel AI SDK and Mastra. Install logs areic-install-*.log.txtin the same folder. - Before we tagged requests per run, we ran one parallel smoke batch whose cost couldn't be split by run. Those logs are kept as
ic-*-smoke.log.txtand aren't counted in the table. - A framework is listed here only after we've run it. Letta isn't listed: it needs its own server, and we haven't set one up yet.