We added TOON, a token-saving alternative to JSON, to the Apify MCP server and ran it in production for 30 days. Two rounds of A/B testing showed no consistent benefit, and token counts showed it made our busiest tool more expensive. So we removed it.
The promise
On October 22, 2025, Johann Schopplich published TOON, the Token-Oriented Object Notation, on GitHub. Its early README claimed "30–60% fewer tokens than JSON." Nobody likes paying for the same JSON keys a thousand times, and within three weeks, the claim spread fast on X, from tutorials to parodies.
Hacker News was lukewarm from the start. The Apify MCP team followed the discussion, and in May 2026 we decided to test TOON on our own data.
Tool results are where the tokens go
An MCP server's job is to feed data to a model. Ours sits in front of Apify Actors, programs used for web scraping and browser automation, so its tools return three kinds of payload: metadata about Actor runs and storages (flat, uniform rows), previews of scraped dataset items (nested objects with optional fields and arrays inside arrays), and the contents of key-value stores. Some responses run to hundreds of kilobytes, and in the clients we evaluated, the text of every response enters the model's context.
We track the size of every tool response, so the cost was visible on a dashboard. TOON looked like the fix.
TOON is a compact, lossless encoding of the same data model as JSON. Its trick is tabular layout: an array of uniform objects declares its keys once, then streams rows like CSV.
Three Actor runs, three ways. First, JSON the way most docs and APIs print it, with indentation and line breaks (191 tokens):
[
{
"id": "VxKZhbtFIKngaNf45",
"status": "SUCCEEDED",
"startedAt": "2026-09-24T18:31:58.951Z",
"usageTotalUsd": 0.011
},
...
]Second, the same JSON with the whitespace stripped. This is what the Apify MCP server sends (143 tokens):
[{"id":"VxKZhbtFIKngaNf45","status":"SUCCEEDED","startedAt":"2026-09-24T18:31:58.951Z","usageTotalUsd":0.011},{"id":"xvXVpNtcMAau9DH8k","status":"SUCCEEDED","startedAt":"2026-09-24T18:31:22.810Z","usageTotalUsd":0.4002},{"id":"JhFJinSOgjDRhR8BD","status":"SUCCEEDED","startedAt":"2026-09-23T13:20:57.100Z","usageTotalUsd":0.0023}]Third, TOON. The field names appear once, then one row per run (123 tokens):
[3]{id,status,startedAt,usageTotalUsd}:
VxKZhbtFIKngaNf45,SUCCEEDED,"2026-09-24T18:31:58.951Z",0.011
xvXVpNtcMAau9DH8k,SUCCEEDED,"2026-09-24T18:31:22.810Z",0.4002
JhFJinSOgjDRhR8BD,SUCCEEDED,"2026-09-23T13:20:57.100Z",0.0023TOON's README headlines 39.9% fewer tokens than JSON, but that JSON is pretty-printed, with indentation and line breaks. The Apify MCP server sends compact JSON, so the number that matters for us is TOON versus compact JSON. There, TOON's own benchmark splits in two. On flat, uniform tables, TOON used 35.3% fewer tokens than compact JSON. On nested or semi-uniform data, it used 14.7% more.
The same README had a section called "When not to use TOON," which said that for deeply nested or non-uniform structures, "JSON-compact often uses fewer tokens."
Three measurements
We tested TOON three ways, all on Apify's own data:
| What was compared | Metric | Data | |
|---|---|---|---|
| Proposal fixtures (May 2026) | the smaller of JSON and dot-flattened TOON, per payload, versus JSON | UTF-8 bytes | 18 payloads, 16 captured from the live API, 2 hand-built |
| Encoder-only (June 2026) | the production encoder (always TOON, dot-flattened, in a markdown fence) versus JSON.stringify | bytes and o200k_base tokens via gpt-tokenizer | fixed payloads: 100 Google Maps places, web pages as text, 10 Actor runs, 4 storage keys |
| Live A/B (June 2026) | the same agent and tasks with TOON on or off, judged by a second model | tool-result bytes, the agent's reported prompt and completion tokens, pass rate | round 1: 20 tasks × 5 runs per format; round 2: 16 storage tasks × 3 runs per format |
The live A/B rounds ran the same tasks through the same agent, Claude Haiku 4.5, once with TOON on and once with it off. For each task we counted the bytes of tool results, the tokens Haiku was billed, and whether it got the answer right. Answers are prose, so a second model, DeepSeek V4 Flash, graded them against a reference answer.
The proposal looked great
Figure 1. Byte change per MCP tool in the May 2026 proposal, which picked the smaller of JSON and dot-flattened TOON for each of 18 payloads. Four list tools shrink 19 to 44%. Dataset items shrink 1.2%, and during the month TOON was live that tool received 99.8% of the calls made to the five tools we changed.
The four list tools came out 19 to 44% smaller. These are TOON's home turf, arrays of uniform metadata objects.
The proposal was careful in a way that matters later: it recommended encoding each response both ways and returning whichever was smaller, so the change could never be worse than the status quo. It also said plainly that the win was measured in bytes while the cost we pay is tokens.
Across all 18 payloads the picker cut bytes by 6.2%. Dataset items, the payloads with real scraped content, shrank 1.2%. We underweighted that number.
The proposal had a second table. Letting the agent ask for three fields instead of about sixty cut a 1 MB Google Maps payload to 12 KB, a 98.8% reduction. Full TOON on the same payload: 14%.
What we released
Response-size telemetry per tool, split into text and structured bytes, went in on June 2, 2026. Before merging, we ran the existing agent task suite against the TOON build and it passed. That was the only check before launch. The byte and token measurements came later.
On June 9, 2026, we released TOON encoding for five tools: Actor run lists, dataset lists, key-value store lists, key-value store keys, and dataset items. We deviated from the proposal in one way. Instead of the adaptive picker, we always encoded those tools as TOON.
One stable shape per tool is easier to debug, and MCP traffic is non-deterministic enough already. But the proposal had measured the picker, not always-on TOON. Its 1.2% saving on dataset items came from choosing JSON for three of the four sampled Actors; it said forced TOON on those was 2 to 4% larger. We shipped the forced case and never re-ran the fixtures.
Only the text representation changed. Programmatic clients read the same JSON object through structuredContent, validated by an outputSchema, both added to the list tools in the same change.
What the measurements said
After launch, the response-size dashboard showed no win, so we measured properly: token counts on fixed payloads, then a live A/B test.
Figure 2. September 2026 re-run: token change, TOON versus JSON, on the same dataset items. Google Maps places cost more at every row count. Web pages stay near zero.
The June token counts turned the story around. For 100 Google Maps places, which we'd assumed was a friendly table-shaped case, TOON cost 11.1% more tokens. The September re-run shown in Figure 2 measured an 11.7% increase. Text-heavy web pages came out about even.
In A/B round one (20 tasks, 5 runs per format), TOON produced 4.8% more tool-result bytes and 0.7% more agent tokens. It passed 64% of tasks against 67% for JSON, a gap that means little when pass counts swung between 8 and 18 out of 20 from run to run. Six of the 20 tasks also never called a TOON-encoded tool.
For round two (16 tasks, 3 runs per format), we added tasks pulling real scraped data and changed the get-dataset-items description to nudge agents to request only the fields they need. TOON came out 1.4% smaller in bytes and 1.7% smaller in agent tokens, and passed 92% of tasks against 88%.
Figure 3. Pass rate per run across both A/B rounds. Results vary more from run to run than between formats.
Every headline number flipped sign between rounds. The engineer who ran the evaluation summed it up: TOON didn't hurt, but it didn't help enough to be worth maintaining.
Why the first benchmark missed
The format did what its documentation says it does. The assumptions were the bug.
The proposal's savings sat on the tools that carry almost no traffic. During the 30 days TOON was live, dataset items received 27.5% of all tool calls to the server and 99.8% of the calls to the five tools we changed. The other four tools, the ones that shrank 19 to 44%, together received 0.07% of all server tool calls. The proposal measured savings per tool and never weighted them by volume.
The dominant payload is nested, and TOON's table needs flat rows. A Google Maps place or an Instagram post has nested objects, optional fields, and arrays inside arrays. When objects aren't uniform, TOON drops the table and writes each one as an indented list, one line per field, so every key is still repeated. On 100 Google Maps places, indentation alone was 60% of TOON's extra tokens, and our dot-flattening made the repeated keys longer.
The README said compact JSON often wins on this kind of data.
Bytes made the picture look better than it was. The proposal measured bytes because tokens depend on the model, and bytes are free to count in production. The June test found that a run list used 9.5% more tokens as TOON, while a September re-run on different data showed a 44% byte saving translating to 23% fewer tokens. Token costs didn't track byte sizes consistently.
We could see which tools were called and how large their responses were. We couldn't see which fields agents used or how nested the payloads were, and we didn't look at the traffic weights we did have until after the release.
The removal
We considered restoring the adaptive picker and returning the smaller encoding per response. We decided against it: a response format that flips with byte counts is confusing for clients and for us, and the measured upside had shrunk to noise.
TOON was in production for 30 days. The removal was small because TOON only ever changed the text output. The structured JSON never changed.
The structured channel and output schemas stayed unchanged; the text serialization returned to JSON.
What survived is the useful part. Byte and token cost metrics are now a permanent feature of the evaluation harness, and the response-size telemetry keeps watching every tool. The change we kept for controlling context size is a simpler one we made at the same time: get-dataset-items returns 20 items by default, and when a dataset is larger than 50 KB, the run and dataset tools tell the agent to narrow with fields= or page with offset.
As of September 24, 2026, TOON has about 25,400 GitHub stars and documentation that describes its limits accurately. It's a good format for flat, uniform arrays. Scraped web content is rarely that.
What another benchmark found
After the removal we found a paper from May 2026. Lorenz Kutschka and Bernhard Geiger published "Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems". They tested TOON and a second format, TRON, inside four agentic benchmarks, single- and multi-turn, with five open-weight model configurations, separating comprehension from generation.
In their benchmarks, TOON reduced token use by up to 18%, with accuracy reductions of up to 9 percentage points compared with JSON.
Their results concern different models and benchmarks from ours, but both studies support evaluating serialization formats against the intended workload.
Takeaways for MCP server maintainers
Some of our tool results reached hundreds of kilobytes. MCP let us keep structuredContent stable for programmatic clients while our own per-call telemetry measured changes to the text representation, which made the change easier to measure and revert. We changed five practices after it.
- Measure tool results in tokens, per tool, with the tokenizer of the model in your evaluation. Byte savings didn't reliably predict token savings across our tests.
- Weight every optimization by traffic. A 44% saving on tools that receive 0.07% of calls is a rounding error. A per-tool response-size counter is a small change that gives you a clearer picture.
- Project fields before you change the encoding. In the proposal's own fixture, projecting three fields cut bytes by 98.8%; full TOON cut them by 14.0%. Item caps and field selection need no new parser on the client side.
- Prefer one predictable response shape per tool, and re-measure whatever you actually ship. Dropping the adaptive picker kept the system debuggable and made the regression visible instead of masked. But we measured the picker and shipped always-on TOON without testing it again.
- Keep the measurement infrastructure after a revert. The experiment cost little and the removal cost less. The cost metrics and telemetry it left behind outlive both.
The evaluation outputs, the parsed data, and the chart code are at github.com/apify/toon-mcp-benchmark, with a manifest tying every file to its upstream commit. The encoder-only test was re-run on fresh data in September 2026 (rerun/ in the repo): 100 Google Maps places still cost 11.7% more tokens as TOON, with TOON 2.3.0 and 4.1.1, the latest release at the time.
