Chapter 7: Temperature, Top-p, and Sampling — The PM Version

Introduction

In Chapter 2, we saw how transformers became Large Language Models.

In Chapter 3, we covered tokens and context windows — the working memory of an AI system.

In Chapter 4, we introduced AI safety, RLHF, and Constitutional AI — why alignment matters and how human feedback shapes behavior.

In Chapter 5, we went deeper into InstructGPT and the RLHF pipeline that turned base models into instruction-following assistants.

In Chapter 6, we compared prompting, fine-tuning, RAG, and tools — the layers that shape what the model knows and how it behaves.

Now we turn to a question that confuses users, frustrates QA teams, and surprises many product managers:

Why does the same prompt sometimes produce different answers?

The answer is not randomness for its own sake. It is sampling — the process by which a language model turns probability scores into the next token, one token at a time. Temperature, top-p, and related controls determine how adventurous or conservative that process is.

This matters for product managers because sampling settings directly affect reliability, regression risk, user trust, evaluation design, structured output stability, and whether your feature feels “consistent” or “creative.” A support bot and a brainstorming assistant should not use the same generation settings — even if they share the same base model.

This chapter explains temperature, top-p, and sampling from a PM lens — not as a math lecture, but as a practical product control framework.

1. Why the Same Prompt Gives Different Answers

A user submits the same question twice. The model gives two different answers. A PM runs the same eval prompt ten times. Three runs pass; seven fail. A demo looks great on Monday and flaky on Tuesday.

This is normal behavior for most LLM products — unless you deliberately configure the system for repeatability.

Common reasons outputs vary

Users often interpret variation as “the AI is confused.” Engineers often interpret it as “expected sampling behavior.” Product managers need a third view: Is this variation acceptable for this feature?

2. From Probabilities to Picks: How Generation Works

At each step, the model does not “know” the full answer upfront. It predicts a probability distribution over possible next tokens, then picks one. That token is appended to the output. The process repeats until the model stops.

A simplified view of one generation step:

This is why small early differences compound. If the model chooses “However” instead of “Therefore” on token three, the rest of the answer may follow a different path.

3. What Is Sampling?

Sampling is the rule for choosing the next token from a probability distribution. Without sampling controls, a model would always pick the single highest-probability token — predictable, but often repetitive and dull. With sampling, the model can explore plausible alternatives.

Sampling is not the same as intelligence. It does not make the model smarter. It changes how the model expresses uncertainty — more conservatively or more creatively.

For PMs, the key question is:

Should this feature explore alternatives, or should it converge on one reliable answer?

4. Greedy Decoding and Deterministic Outputs

Greedy decoding means always selecting the highest-probability token at each step. It is the most deterministic common strategy — though “deterministic” still depends on model version, infrastructure, and whether other randomness exists.

In practice, many API products achieve near-greedy behavior with temperature = 0 (or the provider’s equivalent “deterministic” setting). Some providers also expose an explicit seed parameter for stronger repeatability.

5. Temperature: The Creativity Knob

Temperature scales the model’s probability distribution before sampling. Low temperature sharpens the distribution — top tokens dominate. High temperature flattens it — lower-ranked tokens become more likely.

Temperature does not change what the model knows. It changes how willing the system is to choose less obvious continuations.

Example

Prompt: “Suggest three subject lines for a renewal email.” At temperature 0.2, subject lines may be safe and similar across runs. At temperature 0.9, subject lines may vary widely — useful for ideation, risky if brand tone must stay tightly controlled.

6. Top-p (Nucleus Sampling)

Top-p, also called nucleus sampling, limits choices to the smallest set of tokens whose combined probability mass reaches p. Instead of considering every possible next token, the model samples only from the “nucleus” of plausible options.

Top-p is often more adaptive than top-k because the candidate set size changes with context. When one token is overwhelmingly likely, top-p may effectively behave like greedy decoding. When probabilities are spread out, top-p allows more alternatives.

7. Top-k: A Simpler Filter

Top-k sampling restricts selection to the k highest-probability tokens, then samples from that fixed shortlist. For example, top-k = 40 means only the top 40 candidates are eligible.

Some providers expose top-k; others emphasize top-p. As a PM, you do not need to master every decoding algorithm. You need to know which controls your stack exposes and what behavior they produce in evals.

In most product discussions, top-p and temperature are the primary levers. Top-k matters when your engineering team or model host explicitly recommends it.

8. Deterministic vs Creative Product Modes

Mature AI products often behave like they have modes — even if users never see the underlying parameters.

The mistake is using one global setting for the entire product. A single “temperature = 0.7” default may be fine for open chat and harmful for structured workflows.

9. Reliability, Regression, and User Trust

Users trust AI features when outputs feel stable enough to act on. Sampling instability erodes trust quickly — especially in high-stakes workflows.

How sampling affects trust

Reliability is not only about correctness. It is about predictable behavior under repeated use. A PM should define acceptable variance before launch — not after users complain.

10. When Low Temperature Matters

Use low temperature when the product job rewards consistency over novelty.

Low temperature reduces randomness. It does not eliminate all failure modes. You still need good prompts, grounding, validation, and evals.

• Start with temperature 0 for structured workflows.
• Increase only if outputs feel too repetitive and repetition is a user problem.
• Pair low temperature with schema validation or post-processing checks.

11. When Higher Temperature Helps

Higher temperature is useful when the product goal is exploration, diversity, or fresh wording — not strict repeatability.

The product contract matters. If the UI says “Generate alternatives,” variation is a feature. If the UI says “Get the answer,” variation feels like a bug.

12. Using Temperature and Top-p Together

Temperature and top-p are often configured together. Temperature reshapes probabilities; top-p limits which tokens are eligible. Used well, they balance stability and flexibility.

Do not tune both blindly. Change one variable at a time in evals. Otherwise you cannot explain why behavior improved or regressed.

13. Sampling Across Models and API Versions

Identical sampling settings do not guarantee identical outputs across model families, model sizes, or provider updates. The probability landscape changes when the model changes.

This is why sampling settings should be versioned alongside prompts and eval suites. When you upgrade a model, re-run evals at the intended temperature and top-p — not just once, but multiple times per test case if variation matters.

14. Sampling vs Prompting and Fine-Tuning

Sampling controls how the model chooses among plausible continuations. Prompting and fine-tuning shape what the model is likely to say in the first place. These layers solve different problems.

A common PM mistake is trying to solve a behavior problem by tweaking temperature. If the model consistently misunderstands the task, fix the prompt, context, or training — not randomness alone.

15. Common Product Scenarios

Here is how sampling choices often map to real product features.

Notice that the same product can contain multiple features with different sampling policies. That is normal and desirable.

16. Structured Output and Sampling

Structured output — JSON, tables, fixed field schemas — is especially sensitive to sampling variation. A single different key, trailing comma, or reordered field can break downstream parsers.

Best practice for structured workflows:

• Use low or zero temperature.
• Specify schema clearly in the prompt or use provider schema modes when available.
• Validate output before passing it to business logic.
• Retry with repair prompt only when validation fails — not on every request.

Example: claim summary JSON

Prompt asks for structured output. At low temperature, the model reliably returns parseable JSON:

{
  "claim_id": "CLM-2026-10482",
  "summary": "Hospitalization claim for appendectomy. Discharge summary present.",
  "missing_documents": ["itemized_bill"],
  "recommended_action": "request_missing_docs",
  "confidence": "medium"
}

At higher temperature, the same prompt may produce acceptable content in a broken envelope:

• Different key names (missingDocs vs missing_documents)
• Extra commentary outside the JSON block
• Valid JSON with a different schema shape
• Markdown fences that parsers do not expect

17. Max Tokens, Stop Sequences, and Seeds

Sampling is not the only generation control PMs should know. These adjacent parameters also shape product behavior.

Max tokens is a product decision, not just an engineering safeguard. If users need concise support replies, a tight max tokens limit enforces brevity even when the model would otherwise ramble.

Seeds are valuable for QA and regression testing, but support varies by provider and may not promise identical outputs across infrastructure changes. Use seeds to reduce variance in evals — not as a permanent substitute for validation.

18. Recommended Starting Points

Defaults should be intentional. Use this table as a starting point, then tune with evals.

These are starting points, not universal laws. Your model, prompt, and eval data should drive the final settings.

19. Designing Multi-Output UX

Sampling variation becomes a feature when the UX is designed for it. Poor multi-output UX feels broken; good multi-output UX feels intentional.

Common patterns

UX principles for multi-output features:

1. Label variation as intentional (“Try another version”).
2. Show what changed — not just a new blob of text.
3. Let users accept, edit, or reject an output.
4. Do not regenerate automatically in high-stakes flows.
5. Log which sampling config produced each variant.

20. Sampling and Evaluation

Evaluating an LLM feature once per test case is often insufficient when sampling is enabled. You need to measure both quality and consistency.

Example: a classification feature may score 92% pass@1 but only 78% consistency across five runs at temperature 0.4. That gap is a product signal — not a footnote.

Eval practices for PMs

• Run high-risk prompts multiple times before launch.
• Track format failures separately from factual errors.
• Compare configs side by side: temp 0 vs temp 0.3 vs temp 0.7.
• Include “regenerate” scenarios in eval sets for creative features.
• Version sampling config in eval reports.

21. Should Users Control Temperature?

Most consumer users do not think in terms of temperature or top-p. Exposing raw sliders creates confusion, support burden, and inconsistent outcomes.

A better pattern is user-facing presets that map to backend sampling configs.

Advanced controls can exist in developer settings or admin panels — not in the default consumer path.

22. PM Decision Framework

Use this framework when choosing sampling settings for a feature.

Step 1: Define the product job

Step 2: Choose the minimum sufficient randomness

Step 3: Validate with repeated runs

Before shipping, run representative prompts multiple times. If unacceptable divergence appears, tighten sampling or add validation — do not rely on “usually works.”

23. What PMs Should Not Do

Avoid these common sampling mistakes.

Sampling is a powerful control. Used carelessly, it creates the illusion of progress while leaving the real product problem untouched.

24. Final Mental Model

The model predicts probabilities.
Sampling turns probabilities into picks.
Temperature controls how adventurous the picks are.
Top-p controls how many options are allowed.
Prompting shapes what is likely.
Validation ensures the product can trust the result.

If you remember only one thing from this chapter, remember this:

Match sampling to the job: low randomness for reliability, higher randomness for exploration — and never confuse variation with understanding.

That is the practical PM answer.

Chapter Summary

Closing Thought

Users ask why the AI “changed its mind.” Often the answer is not that the model learned something new overnight. It is that the product allowed a different path through the same probability landscape.

That is not inherently bad. Creativity, brainstorming, and regenerate flows depend on exactly this behavior. But operational AI — routing, extraction, compliance, structured workflows — depends on the opposite.

Product managers do not need to implement sampling algorithms. They do need to decide which features deserve stability, which deserve exploration, and how the UX makes that contract clear.

Get that decision right, and temperature stops being a mysterious engineering knob. It becomes a product design choice — one you can explain, evaluate, and defend.

The next chapter in this module compares pre-training, fine-tuning, and RLHF — how each stage shapes what a model knows and how it behaves before sampling ever enters the picture.

Chapter navigation