Token-Oriented Object Notation (TOON)

A compact, LLM-optimized representation of JSON data. Token-Oriented Object Notation is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow. It's intended for LLM input as a drop-in, lossless representation of your existing JSON.

---

📘 Introduction

Token-Oriented Object Notation (TOON) is a streamlined, human-friendly format that expresses the full JSON data model using far fewer tokens. It merges the indentation style of YAML with the compact, tabular structure of CSV, producing a format that is easy for humans to read and highly effective for Large Language Models (LLMs) to process.

TOON is designed as a drop-in, lossless alternative to JSON in environments where token usage, clarity, and structural predictability matter.

---

🎯 Purpose of TOON

Modern AI systems can handle large inputs, but token cost remains significant. JSON, although widely used, often consumes more tokens than necessary. TOON aims to solve this by providing:

• More compact schema representation
• Clear array structures with declared sizes
• A predictable layout that reduces LLM parsing errors
• Efficient tabular formatting for uniform object arrays

The result is a format that mirrors JSON's structure but is considerably lighter and more model-friendly.

---

🔍 Example: JSON vs TOON

Original JSON

{
  "context": {
    "task": "Cafes we visited last winter",
    "location": "Delhi",
    "season": "winter_2024"
  },
  "friends": ["rahul", "meera", "tanya"],
  "hikes": [
    {
      "id": 1,
      "name": "Brew Garden",
      "distanceKm": 2.3,
      "elevationGain": 0,
      "companion": "rahul",
      "wasSunny": false
    },
    {
      "id": 2,
      "name": "Coffee Street",
      "distanceKm": 1.1,
      "elevationGain": 0,
      "companion": "meera",
      "wasSunny": true
    },
    {
      "id": 3,
      "name": "Mocha Corner",
      "distanceKm": 3.0,
      "elevationGain": 10,
      "companion": "tanya",
      "wasSunny": true
    }
  ]
}

TOON Equivalent

context:
  task: Cafes we visited last winter
  location: Delhi
  season: winter_2024
friends[3]: rahul,meera,tanya
hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
  1,Brew Garden,2.3,0,rahul,false
  2,Coffee Street,1.1,0,meera,true
  3,Mocha Corner,3,10,tanya,true

---

🌟 Key Capabilities

📊 Token Efficiency

TOON offers significant token savings when compared to equivalent JSON structures.

Benchmark results show:

• ~40% reduction in token usage across mixed datasets
• Higher parsing accuracy with LLMs — TOON averaged ~74%, while JSON scored around 70%

This improvement comes from TOON's compact structure and reduced syntactic noise.

🔁 Fully Lossless JSON Mapping

TOON maintains a perfect one-to-one relationship with JSON. Every TOON file can be converted back into JSON:

• Without losing any fields or values
• Preserving all objects, arrays, and primitive data types
• Ensuring deterministic and reversible encoding

🛤️ Explicit Structure for LLMs

TOON includes structural hints that make it easier for LLMs to parse and reconstruct data reliably:

• Array length declarations (e.g., family[3])
• Field headers for uniform object arrays ({id,name,...})
• Indentation-driven nesting similar to YAML

These built-in cues reduce ambiguity and improve model performance.

📐 Minimal Syntax

The syntax avoids unnecessary punctuation such as excessive brackets or quotes.

This balance provides:

• YAML-like readability
• CSV-like compactness
• Cleaner and more predictable formatting

🧺 Optimized for Tabular Arrays

TOON handles uniform arrays of objects exceptionally well.

Such arrays are represented in a table-like row format, resulting in:

• Less repetition
• Reduced token usage
• Cleaner structure for machine processing

🌐 Multi-language Support

TOON is supported across multiple languages, with active or upcoming implementations for:

• TypeScript / JavaScript
• Python
• Go
• Rust
• .NET
• Additional languages in development

This multi-language ecosystem ensures TOON can integrate into various pipelines and workflows.

---

🚫 When TOON Is Not the Best Choice

Although TOON is powerful, some cases benefit more from JSON or CSV:

❌ Deeply Nested Objects

Highly nested or irregular data structures can sometimes be more compact and straightforward in JSON format.

❌ Semi-Uniform Arrays

If array items have inconsistent fields, TOON's tabular format becomes less effective.

❌ Pure Flat Tables

CSV remains more minimal for pure, flat, spreadsheet-like data.

TOON adds lightweight structural markers (like array lengths and field headers) that slightly increase size compared to raw CSV.

❌ Extreme Latency Sensitivity

In performance-critical or on-device environments (e.g., local/quantized LLMs like Ollama):

• JSON may parse faster despite its verbosity
• It's recommended to benchmark both formats for:
  • TTFT (Time To First Token)
  • Tokens per second
  • Total response time

---

🚀 Installation & Quick Start

CLI (No Installation Required)

Try TOON instantly with npx:

# Convert JSON to TOON
npx @toon-format/cli input.json -o output.toon

# Pipe from stdin
echo '{"name": "Tushar", "role": "sde"}' | npx @toon-format/cli

See the CLI section for all options and examples.

TypeScript Library

# npm
npm install @toon-format/toon

# pnpm
pnpm add @toon-format/toon

# yarn
yarn add @toon-format/toon

Example usage:

import { encode } from '@toon-format/toon'

const data = {
  users: [
    { id: 1, name: 'john', role: 'admin' },
    { id: 2, name: 'Doe', role: 'user' }
  ]
}

console.log(encode(data))
// users[2]{id,name,role}:
//   1,john,admin
//   2,Doe,user

Playgrounds

Experiment with TOON format interactively using these community-built tools for token comparison, format conversion, and validation:

• Format Tokenization Playground
• TOON Tools

---

🛠️ CLI

Command-line tool for quick JSON↔TOON conversions, token analysis, and pipeline integration. Auto-detects format from file extension, supports stdin/stdout workflows, and offers delimiter options for maximum efficiency.

# Encode JSON to TOON (auto-detected)
npx @toon-format/cli input.json -o output.toon

# Decode TOON to JSON (auto-detected)
npx @toon-format/cli data.toon -o output.json

# Pipe from stdin (no argument needed)
cat data.json | npx @toon-format/cli
echo '{"name": "Tushar"}' | npx @toon-format/cli

# Output to stdout
npx @toon-format/cli input.json

# Show token savings
npx @toon-format/cli data.json --stats

Tip: See the full CLI documentation for all options, examples, and advanced usage.

---

📋 Format Overview

Detailed syntax references, implementation guides, and quick lookups for understanding and using the TOON format.

• Format Overview – Complete syntax documentation
• Syntax Cheatsheet – Quick reference
• API Reference – Encode/decode usage (TypeScript)

---

🤖 Using TOON with LLMs

TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern. Wrap data in ````toon` code blocks for input, and show the expected header template when asking models to generate TOON. Use tab delimiters for even better token efficiency.

Follow the detailed LLM integration guide for strategies, examples, and validation techniques.

---

📚 Documentation

Comprehensive guides, references, and resources to help you get the most out of the TOON format and tools.

Getting Started

• Introduction & Installation – What TOON is, when to use it, first steps
• Format Overview – Complete syntax with examples
• Benchmarks – Accuracy & token efficiency results

Tools & Integration

• CLI – Command-line tool for JSON↔TOON conversions
• Using TOON with LLMs – Prompting strategies & validation
• Playgrounds – Interactive tools

Reference

• API Reference – TypeScript/JavaScript encode/decode API
• Syntax Cheatsheet – Quick format lookup
• Specification v2.0 – Normative rules for implementers

---

🤝 Contributing

Contributions are welcome! Please feel free to submit issues, feature requests, or pull requests.

---

TOON - Making AI data interchange more efficient, one token at a time.