Chapter 3: Tokens and Context Windows — The PM Version

Introduction

In Chapter 2, we understood how transformers became Large Language Models and why LLMs should be seen as prediction systems, not databases.

This chapter goes one level deeper into something every AI product manager must understand: tokens and context windows.

At first, this may sound like a technical backend topic. It is not.

Tokens and context windows directly influence:

• how much information the model can read,
• how much it can remember during a task,
• how much an AI call costs,
• how reliable the answer is,
• how long an AI workflow can run,
• how well an AI agent can use tools,
• and whether the final experience feels intelligent or confused.

A lot of AI products fail not because the model is weak, but because the product passes the wrong context, too much context, stale context, or no context at all.

1. What Is a Token?

A token is the unit of text that an LLM reads and generates.

People casually say that models read "words," but that is not technically accurate. Models process tokens.

The exact tokenization depends on the model's tokenizer, but the product implication is simple:

2. Why Tokens Matter for Product Managers

Tokens are not just a technical billing detail. They are a product constraint.

Every AI feature has a token economy.

When a user asks a question, the model may receive:

All of this consumes the available context window.

The model does not only process the user's visible question. It processes the full package sent to it — hidden instructions, prior conversation, retrieved files, tool schemas, API results, and expected output space.

PM Takeaway

When you design an AI feature, you are not only designing the UI. You are deciding:

3. What Is a Context Window?

The context window is the total amount of information the model can consider while generating a response.

Think of it as the model's working desk. You can place documents, instructions, user queries, examples, tool results, and prior conversation on the desk. The model can work with what is on the desk.

But if something is not on the desk, the model cannot reliably use it.

The context window is different from the model's training data.

This distinction is critical. The model may have general knowledge from training, but for a specific business task, you need to provide the right working context.

Simple Analogy

4. Context Window Is Not the Same as Memory

This is one of the most common misunderstandings in AI product discussions.

People say: "The model should remember this." But what do they mean by remember?

There are different types of memory in AI products.

The context window is only one layer. It is temporary. Once the conversation changes, gets summarized, or exceeds the limit, earlier information may be dropped, compressed, or no longer visible to the model.

PM Takeaway

Do not use the context window as your product's long-term memory. Use proper storage.

5. How Context Builds Up During a Conversation

In a multi-turn conversation, earlier user messages and assistant responses are usually included again in the next request. That means context grows as the conversation continues.

This is fine for short chats. It becomes risky for long-running workflows.

Why? Because the model may be forced to process:

• old instructions,
• repeated details,
• irrelevant discussion,
• long tool outputs,
• large documents,
• intermediate reasoning,
• duplicated content,
• stale context.

At some point, the model's working memory gets cluttered.

6. Bigger Context Is Useful, But Not Automatically Better

A larger context window allows the model to handle longer prompts, larger documents, bigger codebases, and more complex workflows. That sounds great.

But bigger context is not automatically better.

A large context window gives capacity. It does not automatically create clarity.

PM Analogy

A large desk helps if the right documents are placed neatly. But if the desk is filled with random files, duplicate printouts, old notes, irrelevant emails, and half-read PDFs, the person working on the task may actually perform worse.

Same with LLMs.

7. Context Rot: The Hidden Product Risk

As token count grows, model performance may degrade.

Anthropic refers to this type of degradation as context rot.

In AI products, context rot may look like:

For PMs, context rot explains why long AI workflows can become unreliable even when they technically fit inside the context window.

Example: Claims Review

Suppose you pass:

• full policy document,
• hospital bill,
• discharge summary,
• previous authorization letter,
• email trail,
• insurer guideline,
• claim notes,
• OCR output,
• user instructions,
• processor comments.

The model may have enough token capacity to receive all of it. But will it focus on the right sections? Not automatically.

8. Context Engineering: The New PM Skill

Prompt engineering gets attention, but context engineering is more important for serious AI products.

Prompt engineering asks: What instruction should we give the model?

Context engineering asks: What information should the model see before it answers?

That second question is often more important.

Context Engineering Includes

A PM does not need to implement tokenization logic. But a PM must define the product rules for context.

Example

Weak context design: Send all claim documents to the model.

Better context design: Send the claim type, policy summary, relevant policy clauses, latest bill extract, discharge diagnosis, previous authorization status, missing document checklist, and user's current question. Exclude unrelated email trail unless the user asks for communication history.

9. Token Budgeting: How PMs Should Think

A token budget is a planning tool. It answers: how much of the context window should be reserved for each part of the task?

A simple token budget may look like this:

A common mistake is filling the input context so aggressively that the model has too little room to produce a useful answer.

PM Takeaway

When defining an AI feature, ask:

Token budgeting should be part of AI product requirements.

10. Token Counting: Measurement Before Execution

Token counting is the discipline of estimating how large your prompt is before sending it to the model.

Token count affects:

• cost,
• latency,
• rate limits,
• model routing,
• context overflow,
• prompt optimization.

For a product team, token counting should not be treated as a developer-only diagnostic. It should support product decisions.

If your product has no token visibility, you are flying blind.

11. Context Overflow: What Happens When You Run Out of Room?

Context overflow happens when the input plus expected output exceeds what the model can handle. In product terms, this means the model has run out of working memory.

Bad experience: "Error: context length exceeded."

Better experience: "This document set is too large to review in one pass. I'll summarize the older documents first, then analyze the most relevant sections."

Product Fallback Options

12. Context Management Strategies

For long-running conversations and agentic workflows, context has to be actively managed. There are three important product patterns.

Pattern 1: Summarize

Use when the conversation is long but the full detail is no longer needed.

"Summarize the first 30 turns into user goals, decisions made, unresolved questions, and next actions."

Pattern 2: Select

Use when only some documents are relevant.

Retrieve only the policy clauses related to maternity waiting period instead of sending the full policy booklet.

Pattern 3: Clear

Use when old tool outputs are no longer useful.

Remove raw API responses after extracting the final structured values.

PM Takeaway

Context management should be designed as a lifecycle.

13. Extended Thinking and Context

Some models support extended thinking, where the model can spend more tokens on internal reasoning before giving a final answer.

Where Extended Thinking Helps

Where It May Be Unnecessary

14. Tool Use Adds More Context Pressure

AI agents often use tools. Tools are powerful, but they add context pressure.

A tool-using workflow may include:

• tool definitions,
• tool call request,
• tool result,
• assistant interpretation,
• follow-up tool call,
• additional result,
• final response.

Every tool call creates context overhead.

If your AI agent calls too many tools, keeps too many raw results, or carries unnecessary history, the workflow becomes expensive and fragile.

PM Questions for Tool-Based Products

15. Context Awareness: Models Knowing Their Remaining Budget

Some newer models can be made aware of their remaining context budget during long workflows. This matters because a model that knows its remaining budget can behave more intelligently.

It can decide:

• whether to be concise,
• whether to summarize,
• whether to continue,
• whether to ask for prioritization,
• whether to avoid unnecessary detail,
• whether to save state before context runs out.

PM Takeaway

Future AI products will not only manage user journeys. They will manage context journeys.

For long-running workflows, your system should know:

16. Long Context vs RAG: Which Should a PM Choose?

A common PM question is: If the model has a huge context window, do we still need RAG?

Yes, often you do.

Long context and RAG solve related but different problems.

Product Rule of Thumb

• Use long context when the model genuinely needs to reason across many pieces together.
• Use RAG when the model needs the most relevant pieces from a larger knowledge base.
• Use both when the task needs broad awareness and precise grounding.

Examples

17. How Poor Context Design Creates Hallucination

Hallucination is not only caused by weak models. It is often caused by poor context design.

PM Takeaway

When an AI answer is wrong, do not only blame the model. Ask:

1. Was the right context retrieved?
2. Was irrelevant context removed?
3. Was the source current?
4. Was the context structured?
5. Was the model told which source to trust?
6. Was there enough output budget?
7. Was the task too broad?

18. Context Design for Enterprise Workflows

Enterprise AI products need stricter context design than consumer chatbots. A consumer can tolerate a slightly vague answer. A business workflow often cannot.

Example: Pre-Auth Claim Review

A good context package may include:

19. A PM Checklist for Tokens and Context Windows

Before shipping an AI feature, ask:

Add these questions to your AI PRDs. Tokens and context should not be afterthoughts.

20. Final Mental Model

Tokens are the units the model reads and writes.
The context window is the model's working memory.
A large context window gives capacity, but not automatic accuracy.
More context can help, but bad context can hurt.

For product managers, the real skill is not simply asking: "How many tokens does the model support?"

The better question is: "What context does the model need to complete this task reliably?"

That is the shift. AI product quality depends on context quality.

The best AI products will not dump everything into the model. They will curate, compress, retrieve, prioritize, clear, and preserve context deliberately.

Chapter Summary

Closing Thought

In traditional software, product managers designed screens, flows, fields, and rules. In AI products, product managers must also design context.

That means deciding what the model sees, what it ignores, what it retrieves, what it remembers, what it forgets, and what it should do when the working memory gets crowded.

This is not a small technical detail. This is one of the foundations of reliable AI product design.

A model with a large context window can still fail if the product gives it the wrong context. A smaller model with clean, relevant, well-structured context can often outperform a larger model drowning in noise.

Chapter navigation