If you've ever used an LLM (Large Language Model) API and looked at the token usage for a request, you may have been surprised. You typed a five-word message, but the bill shows thousands of input tokens. This isn't a bug or an overcharge - it's how LLMs actually work under the hood.
This post breaks down what tokens are, how input and output tokens differ, and why your token count is almost always bigger than your message.
What Is a Token, Really?
A token is not a word. It's a chunk of text - often a word, part of a word, a punctuation mark, or even a single character - determined by a fixed vocabulary the LLM learned during training.
Example:
- "Hello" → 1 token (plus a couple of hidden structural tokens the API uses to format messages)
- "tokenization" → 2 tokens (token + ization)
- "don't" → 2 tokens (don + 't)
This matters because it means token count doesn't scale predictably with word count - a paragraph of simple words might use fewer tokens than a paragraph of technical jargon of the same length.
Input Tokens vs. Output Tokens
- Input tokens are everything sent to the LLM before it starts generating a response.
- Output tokens are what the LLM generates back, one token at a time. Each new token is predicted based on everything that came before it, then fed back in to predict the next one, until the response is complete.
This is the easy part to understand. The confusing part is: what actually counts as "input"?
Why a Tiny Message Can Cost Thousands of Input Tokens
Here's the key insight: the message you type is almost never the entire input. In a real request, several other things get bundled in alongside it - most of which you never see.
1. The System Prompt
Every request carries a system prompt: hidden instructions that define the assistant's behaviour, tone, safety guidelines, formatting rules, product knowledge, the current date, and more. Depending on the platform - a raw API call, a chat interface, or a coding tool - this alone can run from a few hundred to several thousand tokens, before your actual message is even added.
2. Tool Definitions and Tool Call Results
If the assistant has tools enabled - web search, code execution, calendar access, and similar integrations - the definition of each tool is sent along with the request. When a tool actually runs, its result is also fed back in as input. A returned search result or code output can add far more tokens than the tool definition itself, especially across several calls.
3. Conversation History
LLMs don't have memory between requests. Every new message resends the entire conversation so far - your previous messages and the assistant's previous replies - as part of the input. Longer conversations naturally make every new message more expensive, even if it's short.
4. Attached Files and Retrieved Content
Any document, image, or piece of context you attach also gets converted into tokens and added to the input, even if you only see a small preview. A short note might add a few dozen tokens, while a long report can add thousands, before the LLM even reaches your question.
The Hidden Cost of "Reasoning" (Output Tokens)
Not all LLMs do this, but some newer models support an internal "reasoning" or "thinking" step before writing the final answer. When active, that internal monologue generates tokens too.
Putting It Together
So when you type "Hello," what's actually sent to the LLM looks more like:
[System prompt] + [Tool definitions] + [Conversation history] + "Hello"
A Simple Example
Say you type "Hello" in an ongoing chat with a couple of tools enabled. Here is roughly how that request breaks down under the hood:
- Prior conversation history: 900 tokens
- Tool definitions and tool call output: 550 tokens
- System instructions: 550 tokens
- Your message ("Hello"): 1 token
Total Input Tokens: ~2,001 tokens
Total Output Tokens: ~150 tokens (e.g., 50 hidden reasoning tokens + 100 tokens for the visible reply)
Your message was one token. Everything else - instructions, tools, tool results, and history - made up the rest. That's the invisible scaffolding every request carries along with it.
The Takeaway
- A token is a fragment of text, not a word.
- Input tokens include your message plus everything bundled with it: system instructions, tool definitions, conversation history, and attached content or tool results.
- Output tokens are what the LLM generates back. On models with reasoning effort enabled, this includes internal reasoning too, not just the final visible reply.
- Longer conversations cost more over time, even with short messages, because history keeps getting resent.
Understanding this makes token usage and API costs far less mysterious - the number was never really about your message. It's about everything travelling with it.
