Tokenize JSON

How to use Tokenize JSON

1. 1. Paste your JSON. Put the JSON document you want broken down into the input pane. It gets scanned character by character the way a JSON lexer would, before any parsing happens.
2. 2. Choose the Output format. Pick One per line for a readable, labeled listing of each token and its type, or JSON array to get the token list back as JSON itself for further processing.
3. 3. Decide on whitespace tokens. Turn on Include whitespace tokens if you want spaces and newlines listed as their own entries, which matters when studying exactly how a lexer treats formatting characters.
4. 4. Copy or review the token list. Copy the resulting list of punctuation, keys, numbers and literals, or read it directly on the page to see exactly how the document decomposes into its smallest pieces.

When to use Tokenize JSON

Tokenize JSON splits a JSON document into the individual tokens a lexer would produce, labeling each one as punctuation, a key, a number, a string or a literal. It is for anyone working at the level below parsing, where you need to see the raw pieces rather than the assembled tree.

• Debugging a hand-written lexer. You are building a JSON parser from scratch and want a reference token stream to compare your own lexer's output against for a tricky input.
• Teaching how JSON grammar works. You want to show a student exactly how a small object with a key and a nested array breaks down into brace, key, colon, bracket, number and literal tokens before it becomes a tree.
• Verifying a syntax highlighter. A custom code editor highlights JSON by token type, and you want to confirm the token boundaries it detects match the actual lexical structure of a sample document.
• Diagnosing a streaming parser failure. A streaming JSON parser chokes on a specific document and you want to see the token-by-token breakdown to find exactly where the stream is going wrong.

Examples

{"a": [1, true]}

punctuation  {
key          "a"
punctuation  :
punctuation  [
number       1
punctuation  ,
literal      true
punctuation  ]
punctuation  }

About the Tokenize JSON tool

Frequently asked questions