Corrections log
When we get something wrong in a test, a grader, a score or a cost figure, we fix it and record it here. We don't quietly edit published results. Found a mistake? Email hello@agentscanr.com. Vendors can also use our right of reply.
| Date | Page | What was wrong | What we changed |
|---|---|---|---|
| 2026-10-14 | CrewAI / Browser Use test | Our first regex grader marked correct answers as failures (for example "INV-103 is not overdue"). | Fixed the grader, checked every output by hand. Build log #2. |
| 2026-10-15 | Leaderboards (round 2) | The grader missed correct answers written on one line (DSPy) or as bare JSON (smolagents). | Fixed the parser and re-graded every run. Build log #3. |
| 2026-10-15 | Inbox triage | Our answer key said a polite return request was "low" priority. Nearly all 190 runs said "normal", and our rubric was ambiguous. | Dropped that check for every run and documented it in the grader. |
| 2026-10-15 | Grok Voice 3.0 | Our running cost ledger first logged voice calls at wall-clock time, lower than the conservative audio-minutes estimate the page reports. | Added a $1.13 ledger entry so the ledger matches the conservative audio-minutes figure. |
| 2026-10-15 | Safety leaderboard and build log #5 | We called the approval policy a "two-sentence rule". The policy added to the system prompt (source) is three sentences: two on approvals and drafting, one on ignoring instructions inside emails and documents. The results were not affected. | Changed the wording to "three-sentence" everywhere it appeared. |
| 2026-10-15 | Asks before acting | Some agents claimed refunds and payments they never made, but passed the "no unsafe tool call" check. | Added a "safe and honest" column, labelled as a post-run addition, and ranked by it. |
| 2026-10-15 | Grok Voice 3.0 | We wrote that xAI's "40% lower latency" claim was "vs Grok Voice 2.0, we assume". xAI's announcement compares 3.0 with GPT-Live-1 (Astra). | Fixed the claims table, then ran 2.0 against 3.0 ourselves (build log #6). |