Token

You might say

Everyone says AI pricing is based on tokens. What is a token, and how does it relate to what I type?

A small piece of text or code that a model reads and writesModels split text, punctuation, and code into tokens using their own rules. Input and output token counts affect available context, wait time, and cost; a token is not always one word or one character.

Know first

AI Application Basics

How a mixed-language sentence becomes tokens

Your inputFix loginButton.tsx

1Fix2fix3 login4Button5.6tsx

Input 6 illustrative piecesOutput 3 illustrative piecesusage = input + output

This illustrates how chunks relate; it is not an exact Token result for any specific model.

How text becomes pieces a model processes

Split the text: The model breaks a sentence into text, punctuation, or code pieces using its own tokenizer.

Read and write: It reads input pieces and produces output pieces one at a time.

Count both sides: Both what you send and what the model returns count toward usage.

Quick checkChoose the best answer

A team estimates context and cost with one Chinese character equals one token, but bills never match. How should usage be estimated?

Keep counting each character, letter, and punctuation mark as one tokenCount only the final AI answer and ignore system rules, history, and filesUse the selected model's tokenizer or actual usage for input, output, and included context

You can say this to an AI Agent

Report input, output, and cached tokens for the current model, including system rules, history, files, and tool results in input. Do not convert directly from character count; explain context usage and the cost basis.

Learn next

1Context Window→2Token Usage & Cost→3TTFT / TPS→

Want GPT or Codex?VibeHubGPT support

GPT-6 Astra

Build in 3D. Put your computer to work.