My LinkedIn feed has turned into a TOON argument. Half the posts treat it like the next standard for AI data formats, the other half call it a badly implemented CSV that is fundamentally broken. The reality is much more boring and much more useful: TOON is a narrow tool that works well in a narrow slice of LLM workflows, and it is not very good outside that slice.
What TOON actually is
TOON, Token-Oriented Object Notation, is a data format designed specifically for large language model prompts and responses. It borrows the idea of a header from CSV and an indentation style similar to YAML, then strips out most of the punctuation that makes JSON noisy for tokenizers.
Here is a simplified example.
JSON:
{ "users": [ { "id": 1, "name": "Alice" }, { "id": 2, "name": "Bob" } ]}
TOON:
users[2]{id,name}: 1,Alice 2,Bob
Same semantics, fewer characters, and usually fewer tokens. TOON declares the schema once and then encodes each row as a comma separated list of values. The keys are implied by position instead of being repeated for every row.
That last part is the entire point. TOON throws away structural redundancy that humans and JSON parsers like, and that LLM tokenizers mostly treat as noise.
Why people care: tokens, cost, and retrieval
Tokens are the real budget for LLM work. If you send the model a giant blob of JSON, you pay for every brace, every quote, every repeated key. That eats into your context window and your wallet without adding much semantic content.
For flat, repetitive data, TOON often cuts token usage by something like 30 to 60 percent compared to JSON. That means you can fit more rows into the same context window, or you can pay less for the same task.
The official TOON GitHub benchmarks and some independent writeups show that, on certain structured retrieval tasks, models not only use fewer tokens but also return more accurate results because there is less syntax noise to wade through. Fewer tokens gives the model more room to focus on actual content instead of parsing braces and quotes.
I originally assumed JSON would be safer because models have seen it everywhere during training. What the benchmarks suggest is that if you give the model a much smaller, cleaner representation of the same data, the token savings can matter more than the familiarity with the format, at least for flat data.
Reported token reduction range for TOON vs JSON on flat, repetitive data.
The attention question: do wide rows become brittle?
One of the better criticisms of TOON came from people asking how it behaves with large or wide rows. In JSON, repeated field names act as structural anchors. The model keeps seeing "user_id", "amount", "timestamp", and so on. That constant repetition reinforces the mapping between names and values.
TOON removes those anchors. The model has to rely purely on positional mapping: column 1 is "id", column 2 is "name", column 3 is "signup_date", and so on. For small to medium objects this is probably fine. For very wide rows with dozens of fields, it is reasonable to worry about misalignment and subtle errors where the model starts answering about the wrong column.
Right now, most of the TOON benchmarks are on fairly friendly structures. I have not seen serious, independent tests on hundred-column analytic tables or extremely wide customer profiles. Until those exist, I would be cautious about pushing TOON into very wide, dense objects where a single column shift is catastrophic.
This is the core trade: you gain token efficiency by removing repeated keys, but you also remove some of the structural grounding that those keys gave the model.
What the structural complaints get right
On LinkedIn, a lot of the strongest pushback framed TOON as fundamentally broken. The list usually looks like this:
- It cannot truly contain null or undefined values.
- It cannot be streamed cleanly.
- It can grow bigger than JSON for complex graph-shaped objects.
- The ecosystem is tiny compared to JSON, CSV, and normal serialization formats.
As a general-purpose serialization format, those are fair points. If you try to use TOON as the storage format for your app, or as an API contract, you are going to have a bad time. That is how you end up calling it fundamentally broken for real-world projects.
But that is also judging it against a job it never signed up for.
TOON as a prompt-only format
When you treat TOON as a prompt-only format, the complaints look different.
- Null and undefined: The model does not care about your exact type system. You can literally send the string
nullorundefinedin a column and the model will understand what you mean. Type fidelity is not the point inside a prompt. - Streaming: TOON is not for long-lived message streams. You generate it, send it, the model reads it, and you are done. If you need durable or streaming transport, you keep that in JSON or another real protocol.
- Complex graphs: If your data is deeply nested or graph-shaped, just keep it in JSON. TOON shines on flat, repetitive structures, not everything.
The way I think about it is simple: your real source of truth stays in JSON or your database schema. TOON is a lossy projection you generate right before sending data to an LLM, then throw away.
In other words: TOON should not be competing with JSON for storage. It should be competing with JSON for the tiny window of time where you are about to pay an LLM to read a chunk of structured data.
Bytes vs tokens and the training data argument
Another thread in the LinkedIn debate is that models are not trained to think in TOON terms, and bytes are not equal to tokens. Both statements are true by themselves, but they do not automatically kill the idea.
TOON is not trying to minimize bytes on disk. It is trying to minimize tokens after the tokenizer has done its work. You can absolutely have a shorter string in bytes that becomes more tokens, but the TOON benchmarks that compare against JSON are counting actual tokens, not characters. That is what matters for cost and context limits.
The training data point is valid too. Models have seen massive amounts of JSON and CSV. They have seen almost no TOON. The mild surprise from the benchmarks is that, for flat structures, the simplicity and size reduction of TOON can outweigh the lack of prior exposure. The model does not need to know the name TOON. It just sees a header and rows of comma separated values with indentation.
Where that argument still concerns me is at the edges: very wide tables, subtle schema drift, and messy real‑world data. This is where I want real A or B tests on my own tasks instead of trusting anyone else’s screenshots.
TOON vs JSON vs CSV: where each one actually fits
Here is a practical way to sort formats for LLM work.
Use TOON when:
- Your data is flat or close to flat, like user lists, logs, product catalogs, simple event streams.
- You have lots of repeated keys in JSON and you are hitting token limits or paying too much.
- You control the pipeline and can easily convert JSON to TOON before the model call and back afterward.
- You care more about model performance and cost than about preserving every detail of the original structure inside the prompt.
Use JSON when:
- You have nested objects, optional fields, and graph‑like relationships.
- You need validation, schemas, and compatibility with existing tools and APIs.
- You want one canonical format for storage, analytics, and inter‑service communication.
Use CSV or markdown tables when:
- Your data really is just a table and humans are going to open it in Excel or a BI tool.
- You want something that every system and every non‑technical teammate can understand instantly.
- You are not yet hitting token or cost issues that justify a new format.
For a lot of standard business workflows, JSON or CSV is still the cleanest choice. TOON only starts to make sense when you are repeatedly shoving large, structured payloads into a model and the token bill or context limit is painful.
Data consistency vs speed
One comment in the thread said the primary goal should be data consistency, and asked who cares how fast it is if data gets lost or mangled. I agree with the underlying instinct. If you mix up identifiers or amounts, you do not get a second chance.
This is where the framing matters. If TOON becomes your primary data representation, you are taking on unnecessary risk for very little upside. If TOON is a throwaway view over a canonical JSON or relational model, you can treat mistakes as one more source of model error that you measure and manage.
I wrote about this more broadly in AI Errors vs Human Errors: You’re Choosing Which Mistakes You Want. You do not get zero mistakes. You choose which type of mistakes you are willing to live with. With TOON, you trade some structural redundancy and familiarity for fewer tokens and potentially better retrieval. That is a good trade in some workloads and a terrible trade in others.
How I treat TOON mentally
I treat TOON like a compression codec for prompts, not like a new foundation for data engineering.
The pipeline looks like this:
- Your app and database speak JSON or whatever you already use.
- Right before a model call, you serialize a subset of that data into TOON.
- You send TOON to the model, get a response, and immediately convert results back into JSON or your domain objects.
Storage formats, API contracts, and analytics do not change. Only the wire format between your application and the LLM changes, and only for specific calls that benefit from token savings.
How to actually test TOON instead of arguing about it
If you want to know whether TOON helps your workflow, do not trust me or the TOON repo or random posts with screenshots. Run a simple A or B test on your own tasks.
- Pick one or two real prompts where you send large, flat JSON payloads into a model.
- Log the current token counts, cost, and accuracy on those tasks.
- Implement a JSON to TOON step right before the model call and a TOON to JSON step after.
- Run the same tasks with TOON several times and compare token counts and outcomes.
If you see lower tokens and equal or better answers, keep TOON for that workload. If you see weird misalignments, worse answers, or no material savings, drop it and move on. You do not owe TOON anything.
Practical rule of thumb
If you want a blunt rule of thumb, here is mine:
- If the payload is small, just send JSON and move on with your life.
- If the payload is large, flat, and repetitive, and you care about cost or are hitting context limits, run a real A or B test: JSON vs TOON on your actual tasks, measuring both token count and accuracy.
- If your data is deeply nested, graph‑shaped, or needs to be streamed, stay with JSON or CSV and ignore TOON entirely.
TOON is not the future of all data formats and it is not a joke either. It is a niche tool for making specific LLM prompts cheaper and sometimes more accurate. If your workload fits that niche, it is worth testing. If not, you are not missing out on some secret weapon by sticking with JSON and CSV.
