How TOON Cuts Token Usage by Up to 60% Compared to JSON for LLMs

Token-Oriented Object Notation, or TOON, handles structured data for large language models in a way that trims token counts. It targets uniform arrays of objects, where JSON repeats field names across rows and adds extra braces and quotes. TOON declares the schema once upfront, then lists the data in a compact, tabular form. This setup cuts overhead for prompts that pack in large datasets.

For automation engineers or anyone feeding data into LLM prompts, this matters because token limits dictate what fits in context. High token use means higher costs or squeezed data. TOON promises 30 to 60 percent fewer tokens in ideal cases, with benchmarks hitting 64.7 percent savings on 50-record sets. That could mean fitting more information without bumping up against API limits.

The format builds on the idea that much of the data sent to LLMs follows predictable patterns, like rows in a table from a database query. Instead of wrapping each entry in its own object with duplicated keys, TOON front-loads the structure. This approach aligns with how tokenizers break down text: fewer unique elements mean less fragmentation into subwords or special characters.

TOON’s Syntax in Action

TOON uses explicit length markers for arrays and a tabular layout. Take an array of users:

users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

Here, users[2] signals two entries, {id,name,role} lists the fields, and the rows follow with commas as delimiters. No quotes around strings unless needed, and indentation handles nesting. You can swap delimiters to tabs or pipes if that shaves more tokens based on the model’s tokenizer.

This contrasts with JSON, which might look like:

{
  "users": [
    {"id": 1, "name": "Alice", "role": "admin"},
    {"id": 2, "name": "Bob", "role": "user"}
  ]
}

The repetition of keys and punctuation adds up quickly in longer lists. TOON avoids that for uniform structures. For nesting, consider a user with addresses:

users[1]{id,name,addresses[1]{street,city}}:
  1,Alice:
    Main St,New York

Indentation keeps levels clear without braces cluttering every line. This mirrors formats like CSV but adds schema for LLMs to reference.

Token Savings Breakdown

To show the difference, consider a sample dataset of items with SKU, quantity, and price. In JSON, it spans more characters due to keys and structure. TOON condenses it to:

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.50

For larger arrays, the gap widens. Here’s a quick comparison on token counts using a common tokenizer like the one in GPT models:

These figures come from benchmarks on datasets like the ones in TOON’s repo. Actual counts vary by tokenizer, but the pattern holds: TOON shines on flat, repeating data. For a 100-record set, JSON might hit 1300 tokens while TOON drops to 450, a 65 percent cut. This scales with repetition; the more rows sharing fields, the better TOON performs.

Tokenizers treat punctuation differently across models. OpenAI’s cl100k_base splits quotes and braces into extra tokens, amplifying JSON’s overhead. TOON’s plain text rows often merge into single tokens, especially for short fields like IDs or numbers.

Features That Drive Efficiency

TOON sticks to indentation for nesting, much like Python code or YAML, which keeps it readable without extra symbols. The strict spec ensures tools like the Elixir encoder hit full compliance, so outputs stay consistent.

Delimiters give flexibility. Commas work for most, but tabs might play better with tokenizers that penalize certain characters less. Explicit lengths, like [2] for two items, help LLMs know when data ends, cutting guesswork in parsing.

For output guidance, the schema upfront acts as a template. An LLM seeing users[5]{id,name} knows to generate five rows with those fields, reducing errors in responses. This self-validation feature turns the format into a prompt scaffold, where the model mirrors the input structure.

Human readability adds another layer. Unlike minified JSON, TOON’s layout makes quick scans easy, useful during debugging in automation scripts. The format avoids ambiguity by requiring uniform fields per array, forcing clean data upfront.

When TOON Fits Best

Uniform arrays make up a lot of LLM inputs in automation: think logs, user lists, or inventory tables. If your workflow pulls from databases with consistent schemas, TOON lets you stuff more rows into prompts without maxing out the context window.

In retrieval-augmented generation, where you inject search results as structured data, TOON could lower costs for repeated queries. For prompt engineering with tabular data, it clarifies structure too. Imagine analyzing sales data: TOON packs 200 rows where JSON fits 120, allowing deeper insights in one call.

Automation pipelines benefit directly. Tools like Zapier or Make.com often handle CSV-like data; converting to TOON before LLM steps saves on API fees. In batch processing, where prompts process chunks of records, the efficiency compounds across runs.

That said, for varying objects or deep nesting, TOON loses ground. If one row has extra fields or sub-objects, the format gets messy. JSON handles polymorphism better there. Hybrid use makes sense: TOON for flat tables, JSON for complex graphs.

Trade-offs to Watch

Adopting TOON means adding encode and decode steps. Libraries exist, like Rust’s serde_toon or Elixir’s toon_ex, but your stack might need updates. Prompt templates have to parse TOON, or you risk feeding garbled data.

Savings aren’t blanket. They hinge on data shape and model tokenizer. Nested JSON might token out the same or better in some cases. For short arrays under 5 items, the schema declaration might even add overhead.

The bigger issue: LLMs train on oceans of JSON. Billions of examples mean models parse it natively. TOON, being new, might trip them up. The model could spend extra tokens figuring out the format, erasing savings. Tests show this risk, especially without fine-tuning.

In workflows, test performance drops. If your LLM hallucinates fields or misreads rows, that’s worse than a few extra tokens. Stick to JSON for critical parses until TOON gains traction. Early adopters report mixed results: solid on simple tasks, shaky on edge cases like escaped delimiters.

Tooling gaps persist. While core languages have support, JavaScript or Python bindings lag. Building custom parsers adds dev time, potentially offsetting short-term savings. Security considerations apply too; TOON’s simplicity might expose injection risks if not sanitized.

Implementation Steps

Start with a simple encoder. For Python, you could adapt from the spec:

• Declare array: name[length]{fields}:
• Indent rows with delimiters.
• For nesting, indent sub-arrays.

Integrate in prompts: “Parse this TOON data: [TOON here]. Output in the same format.” Test with small sets first, measuring tokens via API logs.

For decoding, build a parser or use existing ones. In automation, chain it with data pulls from CSVs or APIs. Convert CSV to TOON by inferring fields from headers, then feed to LLM for summarization.

Available tools: Check the GitHub repo for examples. Rust and Elixir cover core languages; ports to others follow. For Node.js, community forks emerge, but verify compliance. In full pipelines, wrap TOON in a middleware layer that falls back to JSON if parsing fails.

Optimization tips: Profile your tokenizer. Run A/B tests on delimiters—pipes often save on models sensitive to commas in numbers. Monitor output fidelity; if the LLM alters row counts, add explicit instructions referencing the length marker.

Community Buzz and Reality

Excitement builds around TOON for cost cuts in the protocol layer between apps and LLMs. It’s a step toward formats tuned for AI, not general use. Developers in AI ops forums discuss integrations, with early wins in data pipelines.

But hold back on full switch. The JSON bias in training data lingers. Models might generalize, but early tests point to hiccups. Use TOON for uniform chunks where you can afford tweaks, like analytics feeds.

For broader adoption, more tooling and benchmarks help. If token prices keep falling, savings matter less, but context limits stay tight. As LLMs handle longer contexts, TOON’s role shifts to cost control in high-volume apps.

In my setups, I’ve eyed TOON for data-heavy prompts. It works for lists, but I test outputs closely. Pair it with JSON fallbacks for mixed data. Over time, as models see more TOON in fine-tunes, the familiarity gap closes.

TOON vs JSON: Quick Guide

TOON optimizes for LLM inputs with uniform data, but JSON’s ubiquity keeps it default. Test in your pipeline: encode a sample dataset, count tokens, run prompts, check outputs. If savings hold without accuracy loss, integrate. Otherwise, hybrid approaches work—TOON for lists, JSON elsewhere.

This format pushes data handling toward AI-specific needs. As workflows scale, tools like TOON could standardize compact inputs. For now, weigh the gains against integration effort and model quirks. In automation, where structured data drives decisions, even modest savings add up over thousands of calls.

Related reads: For prompt optimization tips, see my post on LLMs calling tools, where structured data plays a key role.