Verdict in one line: LangGraph (langgraph 1.2.15, create_react_agent, no tools) installed and ran without trouble on every attempt. On these single-call tasks it added nothing and lost nothing compared with calling the model directly, so its answers were as good, or as bad, as the model. Score: 74/100.
What we ran
| Board | Runs | Result | Logs |
|---|---|---|---|
| Invoice chasing | 3 | 11/12 checks. 1 of 3 runs fully correct. In run 1 the chaser went to the wrong customer | round 1 logs (lb-langgraph-*) |
| Supplier invoice extraction | 10 | Text fields right every time, totals wrong every time. Same as the no-framework baseline | invoice_extract-langgraph-* in round 2 evidence |
| Inbox triage | 10 | 10/10 perfect | inbox_triage-langgraph-* |
| Weekly cash flow | 10 | 0/10, averaging 4.1 of 21 checks. Same as the baseline | cashflow-langgraph-* |
Model grok-4.20-0309-non-reasoning, temperature 0. Costs come from logged token usage: about $0.0004 to $0.0018 a run (under £0.0014). In a one-run calculator-tool pilot, LangGraph stopped cleanly at its recursion limit after 40 requests ($0.033), with no runaway.
Score (rubric)
We score on the same basis as CrewAI: the invoice-chasing task, plus what the extra boards showed about reliability.
| Criterion (weight) | Score | Why |
|---|---|---|
| Task success (30) | 20 | 11/12 checks on invoice chasing, but only 1 of 3 runs fully correct, and one chaser went to the wrong customer. A wrong-recipient email is a serious error |
| Autonomy & reliability (20) | 16 | Never crashed or needed help in 33 runs, and it respected its step limit in the tool pilot. −4 because its errors are confident and unflagged |
| Setup & ease of use (15) | 14 | One pip install, and pointing langchain-openai at an OpenAI-compatible URL just worked. −1 because you need a second package for the model |
| Value for money (15) | 15 | Open source (MIT), and fractions of a penny per run |
| Safety & control (10) | 5 | Drafting only, no send tools. We didn’t exercise its human-in-the-loop features |
| Integrations (10) | 4 | Verified with xAI (OpenAI-compatible) only. No SMB app connectors tested |
| Total | 74 / 100 |
Caveats
- Most of what these tasks measure is the model. The no-framework baseline scored the same as LangGraph on all four boards. A reasoning model fixed the arithmetic failures (see the boards).
- CrewAI’s 86 came from 3 runs of one task. On the three new boards CrewAI also matched the baseline, including the failures. We’ll re-score both frameworks under a multi-task version of the rubric and say so when we do.
- LangGraph’s main strengths (graphs, state, checkpoints, human approval) weren’t exercised by these tasks.