Benchmarks

Cost per correct answer, by format.

All input and output tokens of a call, priced with every input token uncached, divided by accuracy. Lower is better. Each workbook's questions go in one call.

CSV5,950 in · 2,260 out

baselinecost / correct

Markdown6,935 in · 2,060 out

−1%cost / correct

TOON6,099 in · 2,363 out

+9%cost / correct

Cellprose6,776 in · 1,567 out

−17%cost / correct

Format	Accuracy	Input tokens	Output tokens	Cost per correct vs CSV	Repeat 1	Repeat 2
CSV	97.4%	5,950	2,260	—	—	—
Markdown table	98.3%	6,935	2,060	−1%	−3%	+1%
TOON	93.1%	6,099	2,363	+9%	+13%	+4%
Cellprose v0.2	98.9%	6,776	1,567	−17%	−14%	−19%

Is Cellprose's difference real?

95% paired bootstrap interval of Cellprose's cost per correct answer over each baseline's, resampling the 14 workbooks (10,000 resamples). An interval that crosses zero is reported as no difference.

Against	Difference	95% interval	Verdict
CSV	−17%	−24% to −8%	win
Markdown table	−16%	−22% to −9%	win
TOON	−23%	−33% to −14%	win

Accuracy by task

Where formats differ.

Both models and both repeats together, with partial credit. Every format finds values; the formats differ on telling typed-in numbers from formulas, a task with only three questions in this corpus.

Format	Lookup n = 128	Cell address n = 56	Formula trace n = 84	Impact n = 92	Typed-in vs formula n = 12
CSV	100%	100%	98.6%	97%	68.1%
Markdown table	100%	100%	100%	98%	74.2%
TOON	100%	96.4%	90.5%	94.9%	75%
Cellprose v0.2	100%	100%	100%	97.2%	100%

Agent mode

The file on disk, plus shell tools.

The agent gets a path and a shell instead of the text in its prompt. Figures are relative to the raw .xlsx with tools, on the workbooks run in agent mode.

Model	Setup	Accuracy	Turns	Tokens vs raw .xlsx	Cost per correct vs raw .xlsx
Haiku 5.5	Raw .xlsx with tools	99.8%	3.29	—	—
	Cellprose file with tools	98.9%	2.46	−34%	−27%
	Cellprose text in the prompt	98.9%	1.00	−67%	−55%
Sonnet 5.5	Raw .xlsx with tools	99.7%	2.79	—	—
	Cellprose file with tools	98.8%	2.43	−18%	−19%
	Cellprose text in the prompt	99.7%	1.00	−73%	−66%

Method and limitations

Read before quoting a number.

Every figure is the mean of two repeats run in the same session (r1 and r2); runs from different sessions are never compared. Both repeats are shown above.
Input tokens are priced as uncached. Identical repeated prompts hit the prompt cache and would otherwise look about 90% cheaper.
Claude Haiku 5.5 and Sonnet 5.5 only. Savings may differ for other models and tokenizers.
14 small, clean, generated workbooks; the largest has 150 rows. Real-world workbooks are not in this run.
On Sonnet 5.5 across all workbooks, Cellprose against CSV is within the noise. The clearer gains are on Haiku 5.5 and on the hard workbooks.
In agent mode the Cellprose file costs about one point of accuracy against the raw .xlsx, on trace and impact questions.
How runs are done and scored: the benchmark protocol. Numbers on this page are generated from the result files by benchmarks/site_data.py, and a test checks them against the published headline.

CellproseGitHub ↗

A1fx

ReadyClick a cell · drag the corner handle · arrow keys move · Esc clears