What is a token, really?

Every enterprise budgeting for AI is buying tokens. Token quotas, token prices, token forecasts. And in my experience, almost nobody in those budget conversations can say what a token actually is. Not in the dictionary sense, but in the sense that matters when you’re spending money on something: what am I actually buying? I find the question more interesting the longer I sit with it, because the answer goes further down than most people expect, and some of it is genuinely not understood yet, by anyone.

The surface answer

The standard answer is that a token is a fragment of text. Models don’t read words or letters; they read tokens, pieces produced by a tokenizer that splits text into common chunks. In English, a token works out to roughly three quarters of a word. Common words are often one token, rarer words get split into several, and punctuation and spaces get swept up along the way. A model reads a sequence of tokens in, and writes a sequence of tokens out, one at a time. When a bill says a million tokens, it’s counting these pieces, in and out.

That answer is true, and it’s the one most explanations stop at. It’s also incomplete in a way that matters. It describes the token from the text side, the way you’d describe a kilowatt-hour as “the thing that makes the lights work.” Accurate, and missing the entire machine.

Following the token down

Here is the part I wish more budget owners sat with. Every token a model produces is the output of an enormous amount of arithmetic. To generate one token, the model runs the input through its layers: matrix multiplications, billions of them, across every parameter in the network. Then it does it again for the next token, and the next. Text generation is a loop of very large math problems, each one producing one small piece of language.

That math doesn’t happen in the abstract. It happens on GPUs, physical silicon in physical datacenters, and silicon doing arithmetic draws electricity. Follow the chain all the way down and it’s concrete at every step: a token requires a pass through the model, the pass requires floating-point operations, the operations run on chips, and the chips draw watts and shed heat that other machines spend more watts removing. Somewhere at the bottom of every token is a small, real quantity of energy, plus a share of the eye-watering capital cost of the hardware it ran on, depreciating over its useful life.

Which means token pricing is best understood as a metered abstraction over energy and capital. When you buy a million tokens, you are renting a slice of a datacenter’s power draw and a slice of its depreciation schedule, packaged in a unit that’s convenient to count. I won’t pretend to know the exact energy cost of a token, and you should be suspicious of anyone who quotes one confidently, because it varies enormously with the model, the hardware, the batch, and the serving stack. But the direction is not in doubt. The token has a physical cost floor. It can get cheaper with better engineering, and it has, dramatically. It cannot get free, because arithmetic is not free.

I find this framing clarifying because it kills a lazy mental model: that tokens are “just software” and software costs nothing to run at the margin. Tokens are metered compute. The meter is real.

The part nobody fully understands

Now the humbling part. We can count tokens precisely. We can price them, bill them, and forecast them. What we cannot yet do is fully explain what the model is doing with the computation each token represents.

The situation reminds me of neuroscience. We can measure a neuron firing with beautiful precision. Voltage, timing, frequency, all quantifiable. What the brain is doing with those firings, how spikes become a thought, is still substantially open. AI is in a similar place with its own unit of activity. We know exactly how much computation a token costs. Why the model spends its computation the way it does, which internal circuits activate, and what they represent, is the subject of interpretability research that is real and progressing and very far from done. Researchers can now trace some circuits and name some features inside these models, and the honest summary is still: a large fraction of what happens inside a transformer between token in and token out is unmapped.

I don’t say that to be mystical about it. I say it because intellectual honesty here is useful, not decorative. We are metering and selling, at industrial scale, a unit of activity in systems we can measure much better than we can explain. That should inject some humility into confident claims about what models can’t do, and some caution into confident claims about what they’ll always cost. Both kinds of claims are extrapolations from a partially understood machine.

Which way does the curve bend

So where does the cost of a token go from here? Honestly: both directions at once, and the tension between them is the whole game.

The downward pressure is real and relentless. Each hardware generation does more arithmetic per watt. Inference optimization keeps finding ways to serve the same model for less: better batching, caching, quantization, smarter serving stacks. Distillation compresses the capabilities of large models into smaller ones that are far cheaper to run. And competition among providers pushes savings into prices. The trend line on per-token price has pointed down for years, steeply.

The upward pressure is just as real. Agentic workloads consume an order of magnitude more tokens per task than chat, because agents plan, call tools, read results, and iterate, re-sending context as they go. Reasoning models spend additional tokens thinking before they answer, and the better answers justify the longer thinking. And underneath all of it, datacenter buildout is running into energy constraints. Power, not chips, is increasingly the binding limit, and constrained inputs do not get cheaper on demand.

Put the two together and you get the honest forecast: per-token prices likely keep falling, while per-outcome spend can still rise, because we keep finding reasons to spend more tokens per outcome. Cheaper units, more units. If you manage an AI budget, this is the sentence to internalize, because it means a falling price list tells you almost nothing about next year’s bill.

Spending them well

Which brings the question back from physics to practice. If tokens are metered compute, the interesting variable is not the price of the meter. It is the shape of your consumption.

Consider the least efficient way to buy intelligence for repeated work: a chat window. Every session, a human re-explains the context, re-pastes the material, re-describes the task, and the model re-reads all of it. The same understanding is purchased over and over, and the human’s time is spent purchasing it. For a one-off question this is fine. For work that repeats, it is like hiring a brilliant consultant with amnesia.

The alternative is to build against the API directly, with structure. An agent whose recurring instructions are encoded once instead of re-typed. Context that is loaded when a task needs it rather than pasted every session. Caching so unchanged material isn’t re-processed at full price. The right model tier for each step, saving the expensive model for the reasoning that needs it. Same tokens, same prices, radically more outcome per token, because the consumption is shaped instead of improvised.

That, I think, is where the efficiency frontier actually is. Not waiting for cheaper tokens, but building consumption that wastes fewer of them.

All the way down

You cannot manage what you cannot define. Most organizations are trying to manage AI spend with the surface definition: a token is a bit of text on an invoice. The real definition goes further down. A token is a metered unit of computation, with a physical floor of silicon and watt-hours underneath it, spent by systems we can measure precisely and explain only partially, on workloads we shape well or badly.

Defining the token all the way down to the watt-hour is not trivia. It is where real AI economics starts.