Chapter 5: Cost per Token, Latency, and Model Tier Tradeoffs — The PM Version

Introduction

In Module 01, Chapter 3, we covered tokens and context windows—the working memory of an LLM call.

In Chapter 4, we covered benchmarks and product-specific evals—how to compare models on your tasks, not leaderboard hype.

This chapter connects quality decisions to money and speed. You will not find provider price tables here. Prices change; contracts differ; cached and batch rates have their own rules. What stays useful is the structure of the bill.

Monthly Cost = (Input Tokens × Input Price) + (Output Tokens × Output Price) + (Cached Tokens × Cached Price) + Tool/Infra

Plug in current list prices from your vendor’s pricing page when you build a memo. The PM job is to own the variables—volume, token mix, tier, routing, cache hit rate—not to memorize yesterday’s cents per million.

1. Why Cost Matters for PMs

Traditional SaaS features have near-zero marginal cost per extra click. LLM features do not. Each successful completion burns input tokens, output tokens, and often retrieval, tools, and human review on top.

Cost shows up in four places PMs care about:

• Unit economics — Can this feature survive at expected usage and price?
• Roadmap tradeoffs — A “smarter” default model may erase margin on a high-volume workflow.
• Reliability budgets — Agent loops and long outputs multiply spend per user action.
• Procurement conversations — You need a defendable model, not “we picked the famous one.”

2. Token Types and What Gets Billed

Providers bill by token category. Names differ by vendor; the PM concepts are stable.

Symbolic unit prices (per million tokens or per your vendor’s unit):

Multimodal and long-context models may bill images, audio seconds, or context tiers separately. When you spec a feature, list every metered dimension, not only “GPT call.”

3. Why Output Tokens Matter So Much

Teams optimize prompts for input size—and forget that verbose outputs dominate many bills. P_out is often several times P_in on the same model family.

High-output patterns PMs should flag in reviews:

• “Explain your reasoning in detail” on every ticket
• JSON wrapped in long natural-language preambles
• Agent scratchpads kept in the user-visible thread
• Unbounded “summarize this 40-page PDF” without length caps
• Multiple draft variants per request (A/B at full generation cost)

4. Unit Economics per Call

Define one representative inference call for your workflow:

• T_in = input tokens per call
• T_out = output tokens per call
• T_cache = cached input tokens billed at cache rate (often a fraction of T_in)
• T_in,uncached = T_in − T_cache (when cache accounting splits uncached vs cached)

Cost per call (symbolic):

C_call = (T_in,uncached × P_in) + (T_cache × P_cache) + (T_out × P_out) + C_tool

Measure T_in and T_out from staging logs (tokenizer or provider usage fields). Guessing “about 2k tokens” in a exec deck is how features get underpriced.

5. Monthly Cost Formula

Let N = successful API calls per month (or per business unit). Let overhead C_fixed cover dashboards, support contracts, reserved capacity, and fine-tune hosting.

Monthly Cost = N × C_call + C_fixed

Expanded:

Monthly Cost = N × [(Input Tokens × P_in) + (Output Tokens × P_out) + (Cached Tokens × P_cache) + C_tool] + C_fixed

For multi-step agents, sum C_call across steps or use C_run per user task × tasks per month.

6. Scaling Scenarios: 10k, 100k, and 1M Calls

Use the same C_call; only N changes. This table is structural—replace symbols with priced values from your contract.

Sensitivity knobs (rank these for your feature):

1. ±20% on T_out (verbosity policy)
2. ±30% on T_in (RAG chunk count)
3. Cache hit rate on system prompt (0% vs 70% hit)
4. Share of calls routed to mid-tier vs frontier
5. Agent steps per task (1 vs 4 LLM calls)

7. Model Tiers: Frontier, Mid, and Small

Tiers are product choices, not vanity labels. Map tiers to workflows—not to “we always use the best.”

Relative pricing is usually order-of-magnitude, not 5%: small may be 10–30× cheaper per token than frontier on the same provider stack. Exact ratios belong in your memo with dated P values.

8. Routing and Cascade Architectures

Routing sends each request to a model tier based on rules or a cheap classifier. Cascades try a cheap path first, then escalate only when confidence is low or eval checks fail.

Model routing in the PRD should specify: triggers for escalation, max steps per user task, and eval gates that block silent downgrade of quality.

9. Caching Economics

Prompt caching (provider feature) discounts repeated long prefixes—system prompts, tool definitions, stable policy docs. You pay a cache write cost once, then reduced P_cache on hits.

Break-even sketch: if cache engineering costs E engineer-weeks, compare monthly savings N × T_prefix × hit_rate × (P_in − P_cache) against E.

10. Latency vs Cost

Users feel time to first token (TTFT) and total time to last token. Finance feels monthly tokens. They pull in opposite directions.

Spec latency budgets per workflow tier: e.g. “routing < 300ms,” “user-visible draft starts < 2s TTFT,” “full report < 45s.” Pair each budget with an allowed tier and max tokens.

11. Cost Risks and Failure Modes

• Retry storms — Failed tool calls that re-invoke the LLM with full context each time.
• Context creep — Chat history grows unbounded; every turn re-sends the thread.
• Agent loops — “Try again” until max steps without a stopping rule.
• Free-tier abuse — Power users on unlimited plans burning frontier models.
• Eval gap — Cheap model ships; human rework costs more than inference saved.
• Price list change — No alert when vendor updates P_in/P_out.
• Shadow AI — Teams paste production data into consumer chat tools (different risk, still cost and compliance).

12. Hands-On Exercise

Pick one real or hypothetical AI feature (e.g. internal policy Q&A, code review assist, claims note draft). Complete:

1. Log or estimate T_in, T_out, and tool costs for one happy-path call.
2. Look up current P_in, P_out, P_cache on your vendor site—date-stamp them in a footnote.
3. Compute C_call and monthly cost at N = 10k, 100k, 1M.
4. Propose one routing rule and one output cap that cut cost ≥20% with acceptable eval risk.
5. Write one sentence for finance: “At 100k calls/month we spend approximately ___ assuming ___.”

13. Decision Framework Matrix

Use this when choosing default tier and architecture for a new workflow:

14. Common PM Mistakes

Chapter Summary

Closing Thought

The PM who wins budget and trust shows the spreadsheet: tokens per call, price symbols dated this quarter, three volume scenarios, and a plan when the stress case happens.

Model choice without economics is branding. Economics without evals is false savings. You need both—which is why the next chapter is the capstone memo that ties benchmarks, cost, and risk into one decision document.

Chapter navigation