Verdict in one line: with a hosted model, both frameworks did the job. CrewAI got the credit-control task fully right 3 out of 3 times for about £0.0012 a run. Browser Use read GOV.UK correctly 3 out of 3 times for about £0.012 a run, but only after we changed one setting: its default request parameters are rejected by xAI’s API. Scores: CrewAI 86/100, Browser Use 79/100.
This follows our local-model test, which we left unscored because a 3B CPU model failed the tasks. The tasks here are identical. Only the model changed.
Setup
| Model | grok-4.20-0309-non-reasoning via xAI’s OpenAI-compatible API (https://api.x.ai/v1). We chose it from the /v1/models list as the cheapest current general-purpose text model. grok-build-0.1 is cheaper but aimed at coding. |
| Price (official, docs.x.ai/docs/models, checked 14 Oct 2026) | $1.25 per 1M input tokens, $0.20 per 1M cached input, $2.50 per 1M output (prompts under 200k tokens) |
| Exchange rate | 1 USD = £0.75377, ECB reference rate for 14 Oct 2026 via frankfurter.app |
| Versions | crewai 1.15.28, browser-use 0.13.11, Python 3.12, same virtualenvs as the local test |
| Cost basis | Token counts from each framework’s own usage report, multiplied by the prices above (cost sheet) |
Test 1: CrewAI, UK credit-control task (3 runs)
Same task, data and prompt as before (CSV). The only code change is the LLM(...) line, which now points at xAI with temperature=0 (script). Correct answer: INV-101, INV-102 and INV-105 overdue, £3,150.60 gross overdue, chaser to Tyne Accountants for £1,170.60.
| Check | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| VAT and gross for all 5 | ✅ | ✅ | ✅ |
| Overdue list (101, 102, 105) | ✅ | ✅ | ✅ |
| Total gross overdue £3,150.60 | ✅ | ✅ | ✅ |
| Chaser to Tyne Accountants, INV-105, £1,170.60 gross | ✅ | ✅ | ✅ |
| Any wrong extra figures (e.g. days overdue) | none | none (“44 days”, correct) | none (“44 days”, correct) |
| Wall time | 3.7 s | 3.7 s | 3.7 s |
| Tokens (in / cached / out) | 514 / 128 / 452 | 514 / 512 / 455 | 514 / 128 / 457 |
| Cost | $0.00164 (£0.00123) | $0.00124 (£0.00094) | $0.00165 (£0.00124) |
Logs: run 1, run 2, run 3. Each run made one model request.
Test 2: Browser Use, read GOV.UK VAT rates (3 runs)
Same task text, headless Chrome, max_steps=8, vision off (script). Correct answer: standard rate 20%, reduced rate 5%, as on gov.uk/vat-rates.
First attempt failed 3/3 because of configuration, not the model. Browser Use’s ChatOpenAI sends frequency_penalty=0.3 by default, and xAI returns HTTP 400 “does not support parameter frequencyPenalty”. Each run hit 6 errors and stopped after 5 consecutive failures, with no result (log 1, 2, 3). We found the default in Browser Use’s source and set frequency_penalty=None. That was the only change. Those rejected requests reported 0 tokens, so we’ve counted them at $0.
| Check | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
Called done, is_successful() True | ✅ | ✅ | ✅ |
| Standard rate 20% | ✅ | ✅ | ✅ |
| Reduced rate 5% | ✅ | ✅ | ✅ |
| Steps | 2 | 2 | 2 |
| Wall time | 6.6 s | 7.4 s | 8.8 s |
| Tokens (in / cached / out) | 11,986 / 256 / 632 | 11,946 / 256 / 742 | 11,902 / 256 / 839 |
| Cost | $0.01629 (£0.01228) | $0.01652 (£0.01245) | $0.01671 (£0.01259) |
Logs: run 1, run 2, run 3. Browser Use costs about 10x more per run than CrewAI because it sends the page’s DOM, about 6,000 prompt tokens per step.
Comparison with the free local model (same tasks)
| Qwen2.5 3B local (CPU) | grok-4.20 non-reasoning (hosted) | |
|---|---|---|
| CrewAI checks passed | 1 of 4 | 4 of 4, in 3 of 3 runs |
| CrewAI time | 40.6 s | 3.7 s |
| Browser Use result | never finished (256 s, JSON schema errors) | correct in 2 steps, 3 of 3 runs |
| Cost per run | £0 | £0.001 (CrewAI), £0.012 (Browser Use) |
Scores (rubric)
These scores cover what this test exercised: the framework, run locally, driving one hosted model on one task each. Where a criterion wasn’t exercised, we gave partial credit and said why.
| Criterion (weight) | CrewAI | Browser Use |
|---|---|---|
| Task success (30) | 30: every check passed in all 3 runs | 30: correct in all 3 runs (once configured) |
| Autonomy & reliability (20) | 18: no intervention and no failures. −2 because the totals are free-text arithmetic with nothing checking them | 15: no intervention once working. −5 because the default config failed 3/3 against xAI |
| Setup & ease of use (15) | 14: install about 20 s, one-line model switch. −1 for needing LiteLLM’s openai/ prefix | 11: install about 20 s, used existing Chrome. −4 because the 400 error meant reading library source to find the fix |
| Value for money (15) | 15: open source (MIT), about £0.0012 a run | 14: open source (MIT), about £0.012 a run. −1 for 10x the token use per task |
| Safety & control (10) | 5: the agent only drafted (no send tool). We didn’t exercise an approval step | 5: read-only task. Stopped itself after 5 consecutive errors. No approval gate exercised |
| Integrations (10) | 4: verified with Ollama and xAI (OpenAI-compatible) only. No SMB app connectors tested | 4: verified with local Chrome plus Ollama/xAI only |
| Total | 86 / 100 | 79 / 100 |
Caveats
- On a pure-reasoning task like this, most of the “task success” comes from the model. CrewAI scored well with Grok and badly with a 3B model. To see how much the framework itself matters, we ran the same invoice task through five other frameworks on the same model: see the invoice-chasing leaderboard.
- 3 runs per tool is enough to check consistency. It doesn’t give statistical confidence.
- Browser Use ran at its default temperature. CrewAI ran at temperature 0.
- Total spend for everything on this page and the leaderboard: about $0.084 (£0.064).