Benchmarks
Three experiments with real API calls and real dollars, graded by a judge that never saw which side it was scoring — including the places memory loses.
Results3 experiments
Every run, prompt and answer is on this page — open any section's details.
01Cost
One agent, one repo, 6 questions. The only thing we changed is whether it went looking through the filesystem or asked memory once.
Judged correct: 6/6 with filesystem tools, 6/6with memory — the saving isn't bought with worse answers.
semanticHow does this codebase express form validation and error messages?
7.7 tool calls vs 1
semanticWhich components in this repo support keyboard shortcuts or display key bindings?
4 tool calls vs 1.3
literalHow is the Dialog component structured and what primitive does it wrap?
2.3 tool calls vs 1
semanticWhat components are available for visualizing tabular data or charts?
2.7 tool calls vs 1
literalWhich file defines the cn class-name utility and what does it combine?
1 tool call vs 1 · grep was already cheap here
literalWhat error message does the useSidebar hook throw when called outside a SidebarProvider, and what is SidebarProvider's default value for its `defaultOpen` prop?
1 tool call vs 1 · grep was already cheap here
| Question | fs in-tokens | recall in-tokens | fs turns | recall turns |
|---|---|---|---|---|
| What error message does the useSidebar hook throw when called outside a SidebarProvider, and what is SidebarProvider's default value for its `defaultOpen` prop? | 1,530 | 5,118 | 2 | 2 |
| How is the Dialog component structured and what primitive does it wrap? | 10,907 | 5,070 | 3 | 2 |
| Which file defines the cn class-name utility and what does it combine? | 3,529 | 5,089 | 2 | 2 |
| Which components in this repo support keyboard shortcuts or display key bindings? | 19,836 | 7,477 | 3.3 | 2.3 |
| How does this codebase express form validation and error messages? | 32,266 | 4,815 | 4.7 | 2 |
| What components are available for visualizing tabular data or charts? | 9,188 | 4,723 | 3 | 2 |
Model claude-sonnet-5 · input tokens from Anthropic API usage.input_tokens (exact, real agentic loop) · priced at $3/M · 3-run average · both arms prompt-cached.
02Recall
Four planning conversations, ten messages each — every message delivered as its own session, the way reopening a chat next week actually works. Then we asked for a summary and graded 40 specific facts.
Project kickoff
Trip planning
Kitchen renovation
Product launch planning
“I'd be happy to write this kickoff email, but I don't have any record of what we've discussed—this appears to be the start of our conversation, so I don't have the project details to summarize.”
The control
Run Claude Opus 4.8 against itself — same prompt, same sessions, memory the only variable. It scores 40/40 with memory on and 2/40 with it off. The model was never the problem.
We're kicking off a new project, codename Atlas.
Codename Atlas
It's a mobile app for our client, Northwind Traders.
Client: Northwind Traders
The hard deadline is March 14th — that can't move.
Deadline March 14th
Tech stack is React Native on the front end with a Go backend.
Stack: React Native + Go
Core team is Priya as lead, Marco on backend, and Lena on design.
Team: Priya, Marco, Lena
Budget is capped at $80,000, all-in.
Budget $80,000
Key product constraint: it has to be offline-first.
Constraint: offline-first
We decided to skip Android for v1 — iOS only to start.
Decision: iOS only for v1
The main risk is the payment integration with Stripe.
Risk: Stripe payment integration
Let's schedule a follow-up review for February 28th.
Follow-up Feb 28th
Write the project kickoff email to the team summarizing everything we've discussed.
Gemini + MemoryLayer10/10 facts
Claude Opus 4.80/10 facts
03Grounding
We ingested a real checkout of shadcn-ui/ui and asked 6 questions about files that exist in this tree. An independent judge checked every answer against the actual source.
In this repository, which file defines the `cn` class-name utility, and what two libraries does it combine?
Gemini 2.5 + recall(explore:true)correct
The candidate correctly identifies the file path (with appropriate repo prefix), correctly names the two libraries (clsx and tailwind-merge), and accurately describes how they are combined (clsx then twMerge).
Claude Opus 4.8correct
The candidate correctly identifies the file as lib/utils.ts, accurately names both libraries (clsx and tailwind-merge), and provides the exact implementation shown in the ground truth source code.
Which file holds the Dialog component here, and what primitive does it wrap?
Gemini 2.5 + recall(explore:true)correct
The candidate correctly identifies that the Dialog component is in dialog.tsx and wraps DialogPrimitive.Root, as shown in the ground truth file.
Claude Opus 4.8incorrect
The candidate claims the import is from '@radix-ui/react-dialog', but the ground truth file shows the import is 'Dialog as DialogPrimitive from "radix-ui"' (not '@radix-ui/react-dialog').
Where is the Select component defined in this repo, and what does it use for its trigger icon?
Gemini 2.5 + recall(explore:true)incorrect
The candidate claims SelectTrigger does not explicitly render an icon, but the ground truth shows it explicitly renders a ChevronDownIcon as the trigger icon.
Claude Opus 4.8correct
The candidate correctly identifies that the Select component uses ChevronDownIcon from lucide-react as the trigger icon, and correctly shows it wrapped in SelectPrimitive.Icon with the asChild prop, matching the ground truth file content.
Where is the Tooltip component defined in this repo, and what Radix package does it depend on?
Gemini 2.5 + recall(explore:true)correct
The candidate correctly identifies the file location (with appropriate repo prefix), correctly names the Radix package as 'Radix UI Tooltip', and accurately describes the dependency on TooltipPrimitive.Root from the radix-ui import.
Claude Opus 4.8incorrect
The candidate claims the import is `import * as TooltipPrimitive from "@radix-ui/react-tooltip"` but the actual import in the file is `import { Tooltip as TooltipPrimitive } from "radix-ui"`
What error message does the useSidebar hook throw when called outside a SidebarProvider, and what is SidebarProvider's default value for its `defaultOpen` prop?
Gemini 2.5 + recall(explore:true)correct
Both claims are factually accurate: the useSidebar hook throws 'useSidebar must be used within a SidebarProvider.' (line 44), and the defaultOpen prop defaults to true (line 50).
Claude Opus 4.8correct
The candidate correctly identifies the error message as 'useSidebar must be used within a SidebarProvider.' and the defaultOpen default value as true, both of which match the ground truth source code exactly.
Which file defines the Badge component here, and how are its variants declared?
Gemini 2.5 + recall(explore:true)incorrect
The candidate incorrectly states variants are declared using `cva` within a function called `badgeVariants`, when actually `badgeVariants` IS the result of calling `cva()` directly—it's a constant, not a function.
Claude Opus 4.8incorrect
The candidate's code example shows incorrect variant values (e.g., 'hover:bg-primary/80' and focus styles) that don't match the ground truth file, which has different Tailwind classes and additional variants like 'ghost' and 'link' that the candidate's example omits.
Next
Its memory is. One namespace, every agent, every chat, every teammate.
Get started free