Claude's new tokenizer produced no extra tokens for pure Korean text — the cost after the switch, measured on 10 test texts

Executive summary

The tokenizer changed at 4.7

A tokenizer is the rule that splits characters into tokens, the unit a model processes. Because pricing is per token, when the splitting rule changes, the same text is billed as a different number of tokens.

A new tokenizer arrived with Claude Opus 4.7, and the models after it use the same one. The boundary is 4.7, not the model name — going by name alone, Sonnet 4.6 and Sonnet 5 look like one product line, but their tokenizers differ.

Fable 5.1, Mythos 5.1, Fable 5 and Mythos 5 share the tokenizer introduced in Opus 4.7. Anthropic's documentation states: "A prompt counts the same on all four".

Anthropic gives three figures, and they differ

Look up the multiplier and the official documentation alone gives it three ways. The three do not point to the same value.

The three do not contradict each other. The first two each state one representative value; only the third states a range and a condition. The third is the most useful — because it states that the lower bound is 1x. That means some content does not grow at all.

The models overview's 1.351× is a derived value we calculated from the page's word counts (750,000 → 555,000). It agrees with the same page's Haiku 4.5 entry (200K tokens ≈ 150,000 words), so the page is internally consistent.

We ran our own counts

We followed the procedure Anthropic specifies exactly — count the same request once with the model you use now and once with the model you are moving to, then compare input_tokens. Token counting is not billed.

First, we removed the fixed overhead

The value count_tokens returns contains, besides the text, a fixed overhead that differs by model. The documentation says so too: "Token counts may include tokens added automatically by Anthropic for system optimizations. You are not billed for system-added tokens." When we counted a one-character message, the result ranged from 6 to 11 depending on the model. Left in, this overhead skews the multiplier toward 1, and the shorter the text, the more it skews.

So we counted one request with the text included once and another with it included five times, and divided the difference by 4. The fixed overhead goes into both requests equally, so it disappears in the subtraction. Every value in the table below is a net token count per copy obtained this way.

Measurements for the 10 test texts

ASCII share is the proportion of ASCII characters among non-whitespace characters. Token counts are net values per copy; the old-tokenizer column gave the same value on Haiku 4.5, Sonnet 4.6 and Opus 4.6, and the new-tokenizer column gave the same value on Opus 4.7, Sonnet 5, Opus 5 and Fable 5.1. Measured 2026-09-21.

The lower the Hangul share, the higher the multiplier

Sort the table by multiplier and the ASCII share climbs with it. Across the 10 test texts, the correlation coefficient between the two was 0.899. The four texts written only in Korean were 1.000 without exception, and the three ASCII-only texts were above 1.2 without exception.

A side finding — Korean takes many tokens per character

In Korean, one character is almost one token. The Korean long-form prose we measured was 459 tokens for 439 characters on both tokenizers. The English prose was 386 characters: 97 tokens on the old tokenizer and 127 on the new one. Per character, that works out as follows (derived values).

It is still true that Korean uses far more tokens than English per character. But this change narrowed that gap from 4.16× to 3.18× — not because Korean got better, but because English now takes more tokens.

The two test texts have different content, so this is not a comparison of equal amounts of information. It compares tokens per character, and this table alone does not show that token costs are always higher for a Korean-language service.

Multiply by price and the ranking changes

Multiply the measured token counts by the official prices as of 2026-09-21, and the same model switch gives different results depending on the content. The values below are the change in input cost for sending the same text, and are derived values we calculated.

A negative value means the same text became cheaper to send. Only input is calculated; output tokens and caching are excluded.

How to read it

• Sonnet 4.6 → Sonnet 5 cut the price from $3 to $2, a 33.3% reduction. For the four Korean texts we measured, that cut was realized in full. For the JSON log, only 5.3% of it remained.
• Opus 4.6 → Opus 5 keeps the input price at $5. Going by the price list, cost should not change; for Korean it really does not, and for the JSON log it rises 42.0%. The part of this table that needs the most attention: cost can rise on a switch where the price did not change.
• Haiku 4.5 → Sonnet 5 doubles the price, but the actual cost starts at double and goes up to 2.84×.

Cache-read multipliers also differ by model

Prompt caching stores the processed result of the opening part of a prompt that repeats across requests, and reuses it. Storing it is a cache write; reusing what was stored is a cache read. For a long time, the cache-read price was 0.1× the base input price on every model. That is no longer the case.

Fable 5.1's base input price is $10 per million tokens, so a cache read costs $0.25. Fable 5, from the same generation, charged $1, so on reads alone the price is a quarter. In agent workloads that reuse a long system prompt many times, this line item can make up most of the bill.

Thinking tokens are added to output

Input is not the only thing that changed. Opus 5 and Sonnet 5 run with adaptive thinking even for requests sent without a thinking field. Adaptive thinking is a mode in which the model decides for itself how much reasoning a request needs. On the previous generation, the same request ran without a reasoning step.

Thinking tokens are the tokens a model uses while it reasons internally to produce an answer. They are billed as output tokens even when they are not returned in the response. The migration guide puts it this way — "although per-token pricing is unchanged, a workload that ran without thinking on Claude Opus 4.8 can produce more output tokens per request on Claude Opus 5."

The output price is five times the input price. The extra output cost from thinking being switched on can be larger than what you saved from the lower input price. This change has nothing to do with the tokenizer, so it has to be calculated separately. To go back to the old behavior, pass thinking: {type: "disabled"}, but the documentation notes that in that case the model occasionally emits tool calls as plain text or exposes internal tags.

1M is not the old 1M

The input limit (the context window) is the total number of tokens a model can take in at once. The number is still 1 million, but less text fits inside it.

Anthropic states this conversion in words, so it appears to assume English. The four Korean texts measured in section 4 kept the same token count, so work that sends only that kind of input would reach the limit at the same point as before. For input whose token count grows, context compaction kicks in earlier and fewer files fit in at once. Context compaction shortens the conversation before it reaches the limit, to free up room. The documentation also says to leave more headroom in max_tokens, which sets the output cap, and to re-tune the point at which compaction starts.

On the pricing side, there is also a change in the opposite direction. Claude 4.6 and later models use the full 1-million-token context at the base price, with no long-context surcharge past 200,000 tokens.

What you can do today

Done in order, this takes less than 30 minutes. Token counting is not billed, so you can run it without worrying about cost.

1. Count your prompt on both models. This is the procedure Anthropic specifies — count the same request once with the model you use now and once with the model you are moving to, then compare input_tokens. The documented range is 1–1.35×; our measurement gave 1.000–1.420×. Either way, measuring your own input is more accurate than borrowing a representative value.

   curl -s https://api.anthropic.com/v1/messages/count_tokens \
     -H "x-api-key: $ANTHROPIC_API_KEY" \
     -H "anthropic-version: 2023-06-01" \
     -H "content-type: application/json" \
     -d '{"model":"claude-sonnet-4-6","messages":[{"role":"user","content":"your actual prompt"}]}'
   
   # Same content once more, with the model you are moving to
   #   "model":"claude-sonnet-5"
   # The ratio of the two input_tokens values is your workload's multiplier

   Measure with the whole prompt you actually send, not a single sentence. Only with the system prompt, tool definitions and attached files included do you get the real multiplier.
2. If you are measuring a short prompt, remove the fixed overhead. The response above includes a fixed overhead that differs by model (6–11 when we measured), so the shorter the prompt, the more the multiplier skews toward 1. Put the same text in once and five times and divide the difference by 4, and that fixed overhead cancels out. Every table in this article was obtained this way.

   #!/usr/bin/env python3
   import json, os, urllib.request
   
   KEY  = os.environ["ANTHROPIC_API_KEY"]
   PAIR = ("claude-sonnet-4-6", "claude-sonnet-5")   # (model you use now, model you are moving to)
   TEXT = open("my_prompt.txt", encoding="utf-8").read()
   
   def count(model, text):
       req = urllib.request.Request(
           "https://api.anthropic.com/v1/messages/count_tokens",
           data=json.dumps({"model": model,
                            "messages": [{"role": "user", "content": text}]}).encode(),
           headers={"x-api-key": KEY,
                    "anthropic-version": "2023-06-01",
                    "content-type": "application/json"})
       with urllib.request.urlopen(req, timeout=45) as r:
           return json.load(r)["input_tokens"]
   
   def net(model, text):                      # tokens per copy, fixed overhead cancelled out
       one  = count(model, text)
       five = count(model, "\n\n".join([text] * 5))
       return (five - one) / 4
   
   old, new = (net(m, TEXT) for m in PAIR)
   print(f"{PAIR[0]}: {old:.0f} tokens")
   print(f"{PAIR[1]}: {new:.0f} tokens")
   print(f"ratio: {new/old:.3f}")

   The makeup of this article's 10 test texts is given in the table in section 4, by name, character count and ASCII share. Apply the same procedure to your own prompts as-is.
3. Check whether the two models you are comparing sit on opposite sides of the 4.7 boundary. If they are on the same side, their token counts are the same and you can compare prices directly. If they straddle the boundary, multiply the price by the multiplier first, then compare.
4. Recheck the cache-read price for each model. "10% of input" is no longer a rule that holds for every model. If cache reads are a large share of your bill, this line item alone can flip your choice of model.
5. Check whether thinking is switched on, and re-measure output tokens. Requests that ran without thinking on the previous generation run with thinking on the 5 generation. If you calculate from input alone, you miss the extra output.
6. Re-tune max_tokens and your compaction threshold. If the multiplier you measured in the earlier step was above 1, the same 1 million tokens holds correspondingly less text, and thresholds set on the old basis will trigger earlier. If the multiplier was 1, you can leave them as they are.
7. Throw out budgets and limits built on token counts measured with old models. The documentation warns about this explicitly. Recounting costs nothing.

What this is not yet

• We wrote the test texts ourselves; they are not a real workload. All 10 are short, at 141–559 characters, and prompts in a real service are built differently, mixing system prompts, tool definitions and attached files. Do not reuse the multipliers in the table; measure your own prompts with the procedure in §9.
• The JSON log's 1.420× exceeded the documented upper bound of 1.35×. The documentation says "roughly" and "varying by content", so this does not contradict it, but if you set your budget with 1.35 as the ceiling, it may fall short. We could not determine why this test text came out higher.
• The 0.899 correlation between ASCII share and the multiplier comes from a sample of 10. The direction is clear, but the coefficient is not good enough to predict with. That the three 100%-ASCII test texts differed from one another at 1.227, 1.309 and 1.420 shows that limit plainly. This measurement gives no grounds for saying that the Hangul share determines the multiplier — what we confirmed is only that the two moved together.
• count_tokens values are estimates, as Anthropic states. An actual request can differ by a few tokens. That said, the three old-tokenizer models and the four new-tokenizer models each returned exactly the same values, so within this measurement the values did not waver.
• We did not measure output tokens. The point in section 7, that output tokens grow because thinking is on by default, is taken from Anthropic's documentation and is not our own measurement. Since the output price is five times the input price, this side may move more on a real bill.
• The subtraction that removes the fixed overhead also rests on an assumption. When the same text is joined five times, tokens may come out slightly differently where one copy meets the next. We confirmed that the values are consistent with the overhead measured with a one-character message (6–11), but we have not proved that the subtraction brought the overhead to exactly zero.
• We did not measure models from other vendors. OpenAI and Google tokenizers are outside the scope of this article. We did not check whether the same trap exists there.
• Prices and model specifications are as of 2026-09-21. They change often, so recheck the official documentation when you adopt them.

Sources

We opened and checked every primary source ourselves on 2026-09-21. Prices, model specifications and tokenizers change often, so we give the date we checked.

The measurements in this article use only input_tokens from the /v1/messages/count_tokens response. As the documentation states, this value is an estimate, and an actual request can differ by a few tokens. It may also include tokens Anthropic adds for system optimization, which are not billed.