The Official Site Of NAIT (Native AI Teams)

,

Tokenization pitfalls: why your LLM miscounts characters and money

LEARN · LLMS & TRANSFORMERS

Why tokenization matters more than you think

If you’ve ever asked an LLM to count letters in a word, estimate API cost, or enforce a strict character limit, you’ve probably seen surprising results.

A common example is the classic question:

“How many r’s are in strawberry?”

For years, many language models answered incorrectly. The mistake wasn’t because they couldn’t “see” the word—it was because they don’t naturally process text one character at a time. They process tokens.

Understanding tokenization is one of the quickest ways to become more effective when building with LLMs. It explains:

  • Why prompts cost money.

  • Why context windows fill up unexpectedly.

  • Why character counting often fails.

  • Why string manipulation is harder than it looks.

  • Why multilingual text behaves differently.

This article explores the practical side of tokenization with runnable examples using tiktoken, OpenAI’s tokenizer library.

Tokens are not words—and definitely not characters

Humans naturally think in characters and words.

LLMs don’t.

Instead, text is split into tokens, which are pieces of text learned from large corpora. A token might be:

  • a whole word

  • part of a word

  • punctuation

  • whitespace

  • numbers

  • emoji

  • multiple words together

For example, these are all plausible tokens:

  • hello

  • ing

  • world

  • 2025

  • !

Notice the leading space in " world". Many modern tokenizers intentionally include preceding whitespace because it compresses text more efficiently.

As a result:

  • 1 word ≠ 1 token

  • 1 character ≠ 1 token

  • 1 sentence ≠ predictable number of tokens

Why modern LLMs use BPE

Most production LLMs use variants of Byte Pair Encoding (BPE) or closely related subword tokenization algorithms.

The basic idea is surprisingly simple.

Instead of storing every possible word, the tokenizer learns which byte sequences appear most frequently and merges them into reusable units.

Very common words become single tokens.

Rare words become combinations of smaller pieces.

For example, an uncommon word might split into something like:

intercontinental
→ inter
→ continent
→ al

Meanwhile, a common word like:

computer

may remain a single token.

This gives a good balance between:

  • vocabulary size

  • compression

  • multilingual support

  • robustness to unseen words

Token count is what you pay for

Nearly every commercial LLM API charges based on tokens rather than characters.

Suppose these two prompts contain exactly the same number of characters.

Prompt A:

The weather is nice today.

Prompt B:

Antidisestablishmentarianism appears once.

Prompt B often consumes more tokens because rare words split into multiple subword pieces.

This is why estimating cost from character count is unreliable.

Instead, estimate token count.

Measuring tokens with tiktoken

Let’s see what actually happens.

Install the tokenizer:

pip install tiktoken

Now inspect how text is tokenized.

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

text = "The weather is nice today."

tokens = encoding.encode(text)

print(tokens)
print(f"Token count: {len(tokens)}")

To decode the tokens back:

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

text = "The weather is nice today."

tokens = encoding.encode(text)

for token in tokens:
    piece = encoding.decode([token])
    print(repr(piece))

Example output may resemble:

'The'
' weather'
' is'
' nice'
' today'
'.'

Notice that spaces frequently belong to the following token.

Characters and tokens are completely different measurements

Consider this string:

OpenAI 🚀

Character count:

text = "OpenAI 🚀"

print(len(text))

Token count:

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

text = "OpenAI 🚀"

print(len(encoding.encode(text)))

The numbers are usually different.

Emoji are especially interesting because a single visible symbol can correspond to multiple bytes and multiple tokenization patterns depending on the tokenizer.

Never assume:

1 character = 1 token

Why LLMs struggle with letter counting

Humans see:

strawberry

and mentally process:

s
t
r
a
w
b
e
r
r
y

An LLM often processes something closer to:

straw
berry

or another learned segmentation.

Those internal pieces are optimized for language modeling—not spelling.

Asking an LLM:

Count the r’s.

requires reasoning about characters that may not exist as separate internal units.

Modern reasoning models perform much better than earlier generations because they can explicitly reason about the spelling rather than relying purely on token-level intuition. Even so, spelling and character-level manipulation remain fundamentally different from predicting the next token.

Cherry on the cake: the famous “strawberry” puzzle

One of the most widely shared LLM demonstrations asked a deceptively simple question:

“How many r’s are in strawberry?”

Older models frequently answered:

2

instead of:

3

The example became famous because it exposed a key insight:

The model wasn’t “looking” at the word the way humans do.

It was operating over tokenized representations learned during training.

Today’s frontier models are dramatically better at this task, thanks to improved reasoning and training techniques, but the example remains a memorable illustration of why tokenization matters. It also serves as a reminder that language modeling and character-by-character processing are different problems.

Different languages tokenize differently

English tokenizes efficiently because tokenizers are heavily optimized for common English text.

Other languages may require more tokens.

For example:

Hello!

might consume relatively few tokens.

Equivalent sentences in languages with different writing systems or less frequent byte patterns may require more.

This affects:

  • API cost

  • latency

  • context window usage

When building multilingual applications, always measure token usage rather than estimating it.

Numbers aren’t always single tokens

Developers often assume:

1234567890

is one token.

Usually it isn’t.

Large numbers often split into several pieces.

The same applies to:

  • UUIDs

  • hashes

  • base64 strings

  • long URLs

  • hexadecimal values

This is one reason structured data can consume far more tokens than expected.

JSON is more expensive than many people expect

Compare these two prompts.

Natural language:

Summarize yesterday's sales.

Structured JSON:

{
  "task": "summarize",
  "time_period": "yesterday",
  "department": "sales"
}

The JSON includes:

  • braces

  • quotes

  • commas

  • field names

  • punctuation

All of those require tokens.

Readable machine formats are excellent when structure matters, but they are rarely the most token-efficient representation.

Estimating token counts before sending requests

You can estimate prompt size locally.

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

prompt = """
Write a concise explanation of tokenization.
Include examples.
"""

token_count = len(encoding.encode(prompt))

print(f"Estimated tokens: {token_count}")

This is useful for:

  • budgeting API costs

  • preventing context overflow

  • batching requests

  • monitoring prompt growth over time

Many production systems automatically reject or trim prompts before they exceed model limits.

Practical tips for prompt engineering

Keep these rules in mind.

Measure, don’t guess

Always calculate token counts with the tokenizer that matches your target model.

Budget for the response

The prompt isn’t the only thing consuming tokens.

The generated completion also counts.

If a model supports 200,000 tokens, you still need to reserve room for the output.

Be careful with pasted documents

Large PDFs, logs, and source files expand quickly.

A document that “doesn’t look very long” can consume tens of thousands of tokens.

Compress repetitive data

Repeated boilerplate wastes context.

Instead of repeating the same instructions throughout a conversation:

  • summarize earlier content

  • reference previous sections

  • remove redundant examples

Monitor token usage in production

Track:

  • average prompt size

  • average completion size

  • maximum context usage

  • cost per request

Small prompt improvements often produce noticeable savings at scale.

Common misconceptions

“One token equals one word.”

False.

Common words may be one token.

Rare words may be several.

“Character count predicts API cost.”

False.

Cost depends on tokens.

“Whitespace doesn’t matter.”

False.

Spaces are often embedded directly into tokens.

Changing formatting can slightly change token counts.

“English and Japanese cost the same.”

Not necessarily.

Different languages tokenize differently.

Always measure.

Key takeaways

Tokenization is one of the most important concepts behind modern LLMs.

Remember these principles:

  • Models process tokens, not characters.

  • BPE compresses common text into reusable subword units.

  • Token count—not character count—determines cost and context usage.

  • Letter counting and spelling tasks are fundamentally different from next-token prediction.

  • Different languages and data formats tokenize differently.

  • Use tiktoken to measure prompts instead of estimating.

Once you start thinking in tokens instead of words, many previously mysterious LLM behaviors suddenly make sense—from pricing and latency to context limits and those oddly inconsistent character-counting answers.

Call to action

Install tiktoken, tokenize a few of your real prompts, and compare the token counts with your character counts. You’ll quickly discover where your prompts are spending context—and often find simple opportunities to reduce cost, improve efficiency, and make better use of your model’s context window.

BPE, LLMs, LLMs & Transformers, Tokenization