How AI Agents Consume Tokens During Input and Output Computation

An AI token is a small unit of data or text that a model processes to generate results during training or inference. In a typical conversation, an AI model uses 200 to 2,000 tokens.² Token consumption refers to the total number of tokens a large language model (LLM) reads and writes to complete a task. LLM subscriptions, such as ChatGPT or Claude, bill users based on token consumption, also known as token usage or token spend.

In this article, we will discuss Token Consumption in AI agents in detail. We will first understand why a conversation's cost grows faster than its length. Then, discuss what prompt caching does and does not save, why tool-call loops compound the bill, why output tokens are priced above input tokens, what a shared team plan actually limits, and the levers that bend the curve. Our goal is to help you decide whether your AI agent rollout needs a token cost model before it goes live or not.

How an AI Model Breaks Text into Tokens

A token is nothing but a small piece of text, sometimes a whole word or sometimes a part of one. It strictly comes from a fixed vocabulary the model learned during training. The model never sees letters or words the way we do. It sees a sequence of tokens, and it writes its reply one token at a time.

The splitting or breaking of text into tokens is done by a tokenizer, a small program that ships with the model. Common English words such as "the" or "invoice" usually become one token each. A rarer word is broken into two or three pieces—for example, a product code, a surname, or a field name such as opportunity_stage.³ Punctuation, spaces, and numbers take tokens of their own. As such, the same sentence can cost a different number of tokens on two different models, because every vendor trains its own vocabulary.

For English prose, the working rule is about four characters per token, or about 1.3 tokens per word.¹ So, a 500-word email is roughly 650 tokens. The rule breaks for structured text. JSON, markdown, and code tokenize less efficiently, because every brace, quote, and field name is billed too.¹ A CRM record exported as JSON usually costs more tokens than a paragraph with the same number of characters.

Please note that all figures we discussed are averages, not constants. They change with the tokenizer, the language, and the type of text. Treat them as a planning estimate, and measure the actual count against the model you use before you budget for it.

Input and Output Tokens

Every call to a model has two sides.

• The input tokens are everything the model reads before it answers
• The output tokens are everything it writes back

Both are counted on every call, and the vendor prices them differently. The output is the reply you see, plus any tool call the model writes out in a structured form before the tool runs.¹ Both count as output tokens. In contrast, the input has different components:

• System Prompt: The standing instructions the application sends with every call
• Tool Schemas: The definitions of every connected tool, resent on every call in which the tools are attached
• Conversation History: Every earlier message and every earlier reply, in full
• Tool Results: The data the tools returned on earlier steps
• The New Message: The only part of the input that is actually new

Why output tokens cost more than input tokens

You may be wondering why an output token costs several times an input token when the same model does both. The reason sits in the hardware. The input arrives all at once, so the model processes the whole prompt in one parallel pass, and the processors stay busy. The output is produced one token at a time, and each step has to pull the model's full set of weights through memory before it can write the next token.¹ Most of that step is spent waiting on memory rather than computing, and serving many users together only softens this. That's why the output side is slower per token, and the price reflects it.

One thing most important to realize here is that output does not stay output. The reply the model wrote in the first turn is part of the history it reads in the second. So, every output token is paid once at the output rate, and then again as input on every turn that follows it.

What Is Token Consumption in AI Agents?

Token consumption is the total number of tokens a model reads and writes to complete a task, counted across every call the task takes. It is also known as token usage. You would find it behind every AI invoice, every rate limit, and every context-window error.

It is more than just the length of the question plus the length of the answer, though. The model is stateless: it keeps nothing between two calls, not even the conversation it just had.⁴ As such, the application resends the whole history of conversation, the system prompt, and the tool definitions with every new message. The model has to read all of it again and again before it writes a word. That’s why one can say that the reply they see is the smallest part of what you pay for.

Simply put, a ten-turn conversation costs far more than ten times its first turn, as we will see in the calculation section. Below are the three things that decide how many tokens a task consumes:

• The fixed context that every call carries, meaning the system prompt and the tool schemas.
• The growth per turn, meaning the new message, the reply, and any tool results, all of which stay in the history from then on.
• The number of calls behind each turn, since an agent that uses tools makes several calls to answer one message.

How Do AI Agents Consume Tokens?

A plain chat consumes tokens one turn at a time. Your message goes in, the reply comes out, and the next turn starts from there.

An AI agent, on the other hand, is a model that is allowed to use tools. It can look up a CRM record, search a document store, or call an API, and it decides on its own when to do so. Simply put, one message from you can now set off several calls to the model, and each call is billed in full.

Below is what happens inside one turn of an agent:

• The application sends the model the system prompt, the tool schemas, the conversation so far, and your new message.
• The model reads all of it and, instead of a reply, writes a tool call. The tool call is output.
• The tool runs outside the model, and its result is added to the conversation. Running the tool costs no tokens; reading the result on the next call does.
• The application sends everything again, now with the result attached, and the model reads all of it once more.
• The loop repeats until the model has what it needs, and only then does it write the reply you see.

Two things follow from this loop. First, nothing leaves the context on its own. The result the model read on the second call is resent on the third, the fourth, and every call after it, in full, until the turn ends. Second, the tools are paid for twice over: once as the schemas the model reads on every call, whether it uses the tool or not, and once as the results they return.¹

For example, consider a sales agent asked which open deals have gone quiet. The same prompt calls the CRM for the open deals, then for the last activity on each, then for the owner's notes. That is three calls before it writes a line, and the third call reads both earlier results again on top of the whole history.

How Is Token Consumption Calculated?

Think of the AI as a very smart friend with one strange problem. The moment it finishes answering, it forgets everything. It forgets your question, it forgets its own answer, and it forgets who you are.

So, every time you want to keep on the chat, the app or LLM has to read the whole conversation, right from the start, and then add your new message at the end.

Now the money part. The company that runs the AI charges for every token the AI reads and every token it writes. The tokens it reads are called input, and the tokens it writes are called output. As discussed previously, writing costs more than reading.

So the bill for one message is:

(tokens the AI reads × reading price) + (tokens the AI writes × writing price)

Let us count with an example.

Imagine a chat set up like this:

• The rulebook the app reads to the AI every time (its instructions): 3,000 tokens
• Each message you type: 200 tokens (about 150 words)
• Each answer the AI writes: 400 tokens (about 300 words)

Turn 1: the AI reads the rulebook and your message. That is 3,000 + 200 = 3,200 tokens read. It writes 400.

Turn 2: the app reads the rulebook again, then your first message, then the AI's first answer, then your new message. That is 3,000 + 200 + 400 + 200 = 3,800 tokens read. It writes 400 again.

Turn 3: 3,800 + 600 = 4,400. Every turn adds one message (200) and one answer (400) to the pile, so the AI reads 600 more tokens each time.

At 10 turns, it is 59,000. At 20 turns, only twice as many turns, it is 178,000, which is three times as much. At 40 turns, it is 596,000, more than ten times the 10-turn number.⁵

In mathematics, this is called "growing with the square."

One last thing. In this example, the AI had no tools. When the AI uses tools, like looking up a customer in the CRM, every answer a tool sends back also goes on the pile and gets re-read every turn. Those answers are often longer than what the AI writes, so the pile grows even faster. We will count that in the Tool-Call Loops section.

Prompt Caching

Prompt caching is nothing but a discount for reading the same text twice. When a model processes a prompt, it builds an internal working state for every token it reads. The vendor can keep that state for a while. If the next call begins with the same tokens, the model picks up the saved state instead of reading those tokens again.¹

The word to watch is "exactly." The cache matches from the first token of the prompt and stops at the first token that differs. Everything after that point is read fresh, even if it is identical to last time. As such, the order of the prompt decides how much of it can ever be cached. The stable parts, the system prompt and the tool schemas, belong at the top, and the parts that change belong at the bottom.

Below are the common ways a team breaks its own cache without noticing:

• Editing the system prompt between turns, even by a word
• Adding or reordering the connected tools mid-conversation
• Putting the date, the time, or the user's name at the top of the prompt
• Rewriting or trimming an older message in the history

In the best case, where nothing above the new message changes, the cached part is the whole history and the fresh part is the new message alone.¹ Consider the twenty-turn conversation from the previous section. Of the 178,000 input tokens it reads, nearly all can come from the cache; only the new messages, and the first read of the system prompt, are billed fresh. The input side then costs about an eighth of what it did.

However, the discount changes the price of the pile, not its size. Every turn still resends everything before it, and the cached tokens are still counted, only at a lower rate. Run the same conversation to forty turns, and the cached line still triples. The framework states the result exactly: caching divides the growing term by about ten.¹

Compacting A Chat Session

Please note that the two levers work against each other. Compacting the history rewrites the prompt, so the cache built on the old prompt no longer matches, and the next call is read fresh.¹ Compact on a schedule, at a fixed token budget, and not on every turn. Also, the cache does not last forever; the vendor keeps it for a limited time, so a conversation that pauses for a while may start again at the full rate.

Tool-Call Loops

So far, we have counted the growth from one turn to the next. However, an agent has a second loop inside each turn, and it grows the same way. Every tool round-trip resends the whole context the turn started with, plus every result collected so far in the turn, before the model can decide its next step.

The framework works this out and reaches a familiar shape: the input read across the loop grows with the square of the number of round-trips.^1,6 Let us count it with numbers.

Consider one turn in the middle of a conversation, set up as follows:

• Context at the start of the turn (system prompt, tool schemas, history, the new message): 8,000 tokens
• Each tool round-trip adds a tool call and its result: 2,000 tokens
• The agent needs five lookups from the CRM to answer, then writes the reply

Run one after the other; the first call reads 8,000 tokens and writes a tool call. The second reads 10,000, the third 12,000, and so on, until the sixth call reads 18,000 and writes the reply. That is 78,000 tokens read to answer one message, against 8,000 for a plain reply with no tools.

Halving every result saved 15,000 tokens. Batching the five lookups into one round-trip, so all five run at once, saved 52,000. Why the big difference? A smaller result only shrinks what gets re-read. Fewer round-trips means fewer re-reads of everything, the 8,000-token base context included.¹ That's why the framework puts batching first and shrinking the results second.

Please note that only independent lookups can be batched. If the second lookup needs the first one's result, say the contacts of a deal the agent has not found yet, it has to wait, and forcing the two together breaks the task, not the bill.¹ Sequence the dependent calls, and batch the rest.

Impact of Long Conversations in AI

Everything so far has been about the tokens you are billed for. There is a second cost that never appears as a line on the invoice: memory on the vendor's hardware.

When the model reads a token, it makes a set of notes about it, and it keeps those notes for every token in the conversation so that it does not have to read the old tokens again at every step. These notes are the working state, and they are what the prompt cache from the earlier section reuses. The catch is size. The notes take memory, and the memory grows in a straight line with the length of the conversation: twice the tokens, twice the memory.¹

The framework puts a number on it for a large model of about seventy billion parameters. The model itself takes about 129 GB of memory. At 100,000 tokens of context, the notes for that one conversation take about 262 GB.¹ Simply put, one long conversation needs twice the memory of the model that is reading it.

That's why long conversations are hard to serve. The hardware runs out of memory before it runs out of compute, so every model carries a hard ceiling on its context window, and some vendors charge a higher rate beyond a certain length. Serving many conversations on the same machine at once keeps the hardware busy and brings the vendor's cost down, but it changes nothing in the tokens billed to your conversation.¹

What this means depends on how you buy the model:

• As a Service: the context window ceiling is a hard limit your agent has to stay under, and a shorter context is cheaper for the vendor and for you.
• Hosted Yourself: the architecture and the precision decide how long a context fits on a given machine. Designs that share the attention memory across groups of heads cut it in direct proportion, and so does storing it at lower precision.¹

What a shared team plan actually limits

Many companies buy one team plan and put everyone on it. So, a fair question arises: if ten people share a plan, does the plan somehow share the work between them, so that each person's chats cost less? The short answer is no.

Every conversation is its own private thing. When you send a message, the model sees your chat and nothing else. It never sees a colleague's chat, and your colleague's chat never sees yours. As such, there is nothing to share.

The company's total is simply your tokens plus everyone else's tokens, added up, one conversation at a time.¹ The only part two people can ever share is an identical opening, such as the same system prompt at the top. Everything after it belongs to one person only.

When the plan is shared, three things follow:

• One long agent session can use up the window for the whole team, because an agent's tool loops read tokens far faster than a person types.
• The limit is a ceiling, not a discount. Hitting it usually does not cost more; it stops everyone until the window resets, or until someone pays for extra usage.
• The fix is planning, not sharing. Decide who runs the heavy jobs and when, and give the heaviest agents their own limit or pay-per-token billing instead.¹

Token consumption and your CRM data

For an agent that works on CRM data, the tool results are your records. Every time the agent looks up a contact, a deal, or a case, the record comes back as text and joins the conversation. The same gets re-read on every round-trip after it. As such, the size and the shape of your data decide a large part of the bill.

The usual suspects are few:

• Duplicate records, so the agent reads the same customer two or three times
• Records with hundreds of properties, most of them empty, that come back whole
• Long free-text notes where a picklist (a dropdown with fixed choices) would do
• Old records with no owner, which the agent has to read through to rule out
• Tools that return the entire record when the agent asked for one field

The mechanism is simple. The agent trusts the tool; the tool returns the record as it is. For example, a contact with 180 properties, most of them blank, can run past 1,500 tokens as JSON, because every property name is billed even when its value is empty. The five fields the agent needed would be under 100. Now multiply the gap by every lookup, and again by every round-trip that re-reads it.

Clean CRM data, for an agent, means four things in practice:

• One record per customer, with the duplicates merged before the agent goes live
• Picklists and short fields for the values the agent needs, and free text only where a person reads it
• Tools that return the fields the agent asked for, and not the whole record
• One owner per record, so the agent finds the right person in one lookup

This is the same rule Vegacompass applies to every CRM project: fix the data before switching on the automation, because an agent reads whatever the data holds and pays for all of it. The published case studies on Vegacompass for FIMC Insurance and Linq describe legacy data cleanups of exactly this kind.

Quick Tips to Keep Your AI Agent's Token Bill Down

At Vegacompass, we say an agent's bill does not grow because the model is expensive; instead, it grows because nothing is ever taken out of the conversation.

1. Put the stable content first and never edit it

The system prompt and the tool schemas go at the top, the parts that change go at the bottom, and nobody edits the top in the middle of a conversation. One changed word above the new message, and everything below it is read fresh at the full rate.¹

2. Compact on a fixed token budget, not when the window is full

Pick a number, well under the context window, and when the history crosses it, summarize the old turns into a short note and continue from the note. Do it on a schedule, not on every turn, because each compaction throws away the cache. This is the only lever that changes the shape of the growth, and the note must keep what the agent still needs.¹

3. Batch the tool calls that do not depend on each other

If the agent needs the deal, the account, and the last three activities, and none of them depends on another, ask for all of them in one round-trip. Five lookups run one after the other cost three times what the same five cost in one batch, as we counted above.

4. Cap the output and give the agent a place to stop

Set a maximum reply length, a clear stop point, and a lower reasoning effort for routine tasks. Output is the costliest token there is, and every output token is read again as input on every turn that follows it.¹

5. Disconnect the tools the agent does not use

Every connected tool adds its definition to every call, whether the agent uses it or not. Connect the three tools the agent works with, not the twenty the workspace offers, and review the list whenever the agent's job changes.¹

6. Return fields, not records, from the CRM tools

Build the CRM tool so that the agent can ask for five fields and receive five fields. A whole contact record with 180 properties costs many times the fields the agent needed, and it costs that on every round-trip that re-reads it.⁷

7. Book a CRM audit before the agent goes live

An audit reads the data model, the automations, the data quality, and the license usage, and it ends in one written diagnosis with the fixes ranked by the value they return. It is generally a two-week engagement, and the token cost model from the framework is built on what it finds. The right times are before the agent goes live, before it is rolled out to a second team, before the plan renewal, and whenever the token bill and the number of users stop agreeing with each other. To learn more, please read our CRM Audit & Technical Debt page.

On the other hand, an agent that answers in two turns, uses one tool, and is switched off after the session does not need any of this; it needs an output cap and nothing more. At Vegacompass, the senior consultant who writes the cost model is the one who builds the agent on it. Our goal is to ensure that the estimate is written by someone who has to hit it.

Download the framework

Download our framework to bring your token consumption down. It works for any model, bought or self-hosted, because it is written from first principles and names no vendor. Every formula is in closed form, with every symbol defined before it is used, so you can put in your own numbers and price an agent before it goes live.

It covers the two places the bill hides: the tool loops inside a turn and the shared plan across a team, and its levers table tells you which fixes only lower the bill and which ones change its shape. It would help your agents consume fewer tokens, stay within the plan, and cut the bill.

[Download Button]

FAQs

Q. How many tokens are 1,000 words?

About 1,300 tokens for plain English text, using the rule of 1.3 tokens per word.^1,8

The count rises for names, code, and structured formats such as JSON, because every symbol and field name takes tokens of its own. The rule also differs a little from one model to another, since every vendor trains its own tokenizer. For a real budget, measure the text with the token counter the vendor provides, and not with a word count.

Q. Does prompt caching make a long conversation cheap?

Cheaper, not cheap. Caching bills the re-read part of the conversation at about a tenth of the fresh rate, which is a large saving on every turn.^1,9

However, the conversation still grows in the same way, and the cached tokens are still counted.

Q. Do users on the same team plan share tokens?

No. Every conversation is separate, and the model never sees two users' chats together, so the company's usage is simply everyone's tokens added up.¹

What the plan shares is a limit: a set number of tokens or messages in a fixed period that everyone's usage counts against.

References

1. (2026, August 24). A Mathematical Framework for Token Consumption in Transformer Inference and Agentic Workflows. Vegacompass. [download page URL]
2. (2026). OpenAI API Pricing: Full Breakdown of Costs (Jul 2026). Puter Developer. https://developer.puter.com/tutorials/openai-api-pricing/
3. (2023). Tokenization — Large Language Models. https://uhasker.github.io/large-language-models/chapter4/tokenization.html
4. (2026). Troubleshooting API rate limits and 429 errors. OpenAI Help Center. https://help.openai.com/en/articles/5955604
5. (2024). OpenAI Help Center: What are tokens and how to count them?. OpenAI Help Center. https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
6. (2026). Agentic Loop: How AI Agents Perceive, Think, and Act. OpenLegion. https://www.openlegion.ai/en/learn/agentic-loop
7. (2026). Prompt Caching: Cost & Performance Analysis Across Providers. Artificial Analysis. https://artificialanalysis.ai/models/caching
8. (2023). What Are Tokens in LLMs and Why They Matter. LLM Guides. https://llmguides.ai/learn/what-are-tokens/
9. (2026). Prompt Caching: How It Works and What It Saves. LLM Cost Lab. https://llmcostlab.com/reduce-llm-api-costs/prompt-caching/