On 14 October we ran three UK small-business tasks through 14 agent frameworks and several Grok models, 10 runs each, with answers computed in code before any agent saw the task: supplier invoice extraction, customer-service inbox triage and a weekly cash flow from a bank CSV. That’s 570 graded runs, plus the earlier invoice-chasing board. Every log is published.
The pattern was so consistent that it’s the main thing we’d tell anyone choosing an agent stack.
Finding 1: on single-call tasks, the framework was a wrapper
On invoice extraction, every framework running the non-reasoning model (grok-4.20-0309-non-reasoning) got 0 perfect runs out of 10: CrewAI, LangGraph, LlamaIndex, Haystack, AG2, Pydantic AI, the OpenAI Agents SDK, Agno, Strands, smolagents, Google ADK, the Vercel AI SDK and Mastra. So did a single plain API call with no framework. On the cash-flow task it was the same: 0/10 for every one of them.
They didn’t just fail together; they mostly failed the same checks. That’s what you’d expect if the framework is passing the prompt to the model and handing back the answer. For a task that’s one model call, the framework adds code, dependencies and a few tokens of its own prompt template. It doesn’t add accuracy.
Finding 2: letting the model think changed everything
On the same invoice task, every reasoning model got 10/10: grok-4.3, grok-4.20-0309-reasoning, grok-build-0.1 and grok-4.7, all with no framework at all. On cash flow, grok-build-0.1, grok-4.20-0309-reasoning and grok-4.7 were 10/10 and grok-4.3 8/10.
The one framework that broke the pattern proves the point. DSPy’s ReAct module makes the model write a reasoning step before it answers. On the non-reasoning model, it scored 10/10 on invoice extraction and 9/10 on cash flow, where every other framework scored 0.
Finding 3: thinking costs money and time, so spend it where it matters
The chart above plots each entry’s average cost per run against how often it was perfect. The cheap cluster at the bottom-left is the frameworks on the non-reasoning model: about $0.0004 to $0.001 per extraction run, and wrong every time. The reasoning entries cost about $0.002 to $0.012 per run, roughly 4 to 26 times more, and were right every time.
Inbox triage was the counter-example. All 19 entries got every check right in all 10 runs. Paying for reasoning there only bought slowness: the cheapest entries cost $0.0007 per run, grok-4.7 cost $0.0083.
What we’d do with this
- Pick the framework for its tools, not its accuracy. Choose it for orchestration, tool calling, tracing, retries and your team’s language. Don’t expect it to make a model smarter on a single call.
- Test reasoning on, then off. If a task involves arithmetic, totals or reconciling numbers, start with a reasoning model and only drop down if a cheaper one passes your own checks.
- Use cheap models where the task is classification. Triage was solved by everything.
- Measure it yourself. Our tasks are small and synthetic. Run your own task 10 times, not once: several entries were right in some runs and wrong in others.
Limits
Three single-call tasks, one model family (xAI Grok), 10 runs each, temperature 0, run on 14 Oct 2026. Multi-step, tool-heavy work is where frameworks should matter more; our safety test is a first step there. Costs come from our logging proxy at xAI list prices (ledger).