Free lesson · AI Engineering

Understanding LLM Parameters

Easy10 min readaillmrag

Understanding LLM Parameters

The Dial Nobody Understands and Everybody Turns

Here is a pattern I have seen at least a dozen times, at Meta and everywhere else: an engineer's LLM feature is producing inconsistent output. Somebody on the team says "try lowering the temperature." They set temperature=0, the output gets more consistent, the ticket gets closed, and nobody ever thinks about it again - until six months later when the same feature starts producing weirdly repetitive, stilted responses in a different context, and nobody remembers why the temperature is zero or what it even does.

Generation parameters are the most-touched, least-understood part of the LLM API. Engineers cargo-cult settings from blog posts and Stack Overflow answers without a mental model of what the parameters actually control. That works fine until it does not, and when it does not, you cannot debug what you do not understand.

This lesson gives you the mental model. It is one idea, applied consistently: every generation parameter is a modification to a probability distribution. Once you see the distribution, every parameter becomes obvious, every blog-post recommendation becomes checkable, and every weird output behaviour becomes diagnosable.

The One Mental Model: The Next-Token Distribution

Recall from the last topic: the model generates one token at a time. Here is what happens at each step.

The model looks at everything so far - your system prompt, the user message, the tokens it has already generated - and produces a score for every token in its vocabulary (roughly 100,000 tokens for modern models). Those scores get converted into a probability distribution. Something like this, for the prompt "The capital of France is":

Then the model samples from this distribution - picks one token, weighted by probability - appends it, and repeats. That is the entire generation loop.

Every parameter in this lesson does exactly one of two things:

1. Reshapes the distribution before sampling (temperature, penalties)
2. Truncates the distribution before sampling (top-p, top-k)

Plus two that operate outside the distribution entirely (max_tokens, stop sequences), which end generation rather than shaping it.

Hold onto the table above. We are going to modify it repeatedly.

graph LR
    A[Context so far] --> B[Model computes scores<br/>for all ~100K tokens]
    B --> C[Temperature<br/>reshapes distribution]
    C --> D[Top-p / Top-k<br/>truncates distribution]
    D --> E[Sample one token]
    E --> F{Stop token or<br/>max_tokens hit?}
    F -->|No| A
    F -->|Yes| G[Return response]

This flowchart shows the token generation loop. The context so far feeds into the model, which computes scores for all roughly one hundred thousand vocabulary tokens. Temperature reshapes this distribution, then top-p or top-k truncates it, then one token is sampled. If a stop token or the max token limit is hit, the response is returned; otherwise the loop repeats with the new token added to the context.

Temperature: The Sharpness Dial

Temperature is the most famous parameter and the most misdescribed. The popular framing - "temperature controls creativity" - is a lie-to-children that eventually costs you. Here is what it actually does.

Before sampling, every token's score is divided by the temperature value. Then the scores are converted to probabilities. The mathematical effect:

Low temperature (toward 0) sharpens the distribution. High-probability tokens get relatively more probable; low-probability tokens get crushed toward zero. At temperature=0, sampling effectively becomes "always pick the single most likely token" - called greedy decoding.

High temperature (toward 1 and above) flattens the distribution. The gap between likely and unlikely tokens shrinks, so unlikely tokens get sampled more often.

Watch what happens to our example distribution:

At temperature 0.2, "Paris" is essentially guaranteed. At temperature 1.5, there is a nearly 3-in-10 chance the model says something other than "Paris" - and the sentence goes somewhere else entirely from that token onward.

This is why "creativity" is the wrong word. High temperature does not make the model more creative. It makes the model more willing to pick lower-probability tokens. Sometimes that reads as creative. In factual, structured, or code contexts, it reads as wrong.

The Settings I Actually Use in Production

Let me be concrete, because this is where blog posts hand-wave:

Warning: Temperature 0 does not guarantee deterministic output. This surprises everyone the first time. Even at temperature 0, the same prompt can produce different responses across calls - because of floating-point non-associativity on GPUs, batching effects in the provider's inference stack, and model updates behind the same model string. Temperature 0 makes output much more consistent, not deterministic. If your architecture depends on exact reproducibility of LLM output, your architecture is wrong, and we will fix it in the evaluation and caching modules.

And one production war story: we once traced a subtle quality regression in a summarisation feature to a well-meaning engineer who raised temperature from 0.3 to 0.9 "to make the summaries less dry." The summaries did get livelier. They also started occasionally inventing details that were not in the source - because higher temperature means more mass on lower-probability tokens, and in a summarisation context, lower-probability tokens are often things the source did not say. Hallucination risk is not independent of temperature. Remember that connection; it comes back in the RAG and safety modules.

Top-p: The Cutoff Dial

Top-p (also called nucleus sampling) truncates the distribution instead of reshaping it. top_p=0.9 means: sort tokens by probability, keep the smallest set whose cumulative probability reaches 90%, throw away everything else, then sample from what remains.

The elegance of top-p is that it adapts to the model's confidence:

• When the model is confident ("The capital of France is…"), the top token alone might cover 92% - so top-p=0.9 keeps just one or two candidates. Effectively greedy.
• When the model is uncertain ("My favourite colour is…"), it might take 30 tokens to reach 90% cumulative probability - so all 30 stay in play.

Top-p says: sample freely among plausible options, but never sample from the garbage tail. That tail - the thousands of tokens with tiny but nonzero probability - is where the truly bizarre outputs come from at high temperature. Top-p amputates it.

Top-k is the blunter cousin: keep exactly the k most probable tokens regardless of their probabilities. top_k=40 keeps the top 40 whether the distribution is sharp or flat. Most modern APIs favour top-p; Anthropic exposes both; OpenAI exposes top-p only. In practice, top-p is what you will use.

How Temperature and Top-p Interact

They stack: temperature reshapes, then top-p truncates the reshaped distribution. Which produces the classic footgun:

Danger: Do not tune temperature and top-p simultaneously. Both providers say this in their docs and everyone ignores it. If you lower both, you double-constrain toward repetitive output. If you raise both, you get variance from two sources and can no longer attribute behaviour changes to either dial. Pick one as your primary control - I use temperature, leaving top-p at its default (usually 1.0) or at a mild 0.9 safety cutoff - and touch the other only with a specific, testable reason.

Frequency and Presence Penalties: The Repetition Dials

Two parameters modify the distribution based on what has already been generated:

Frequency penalty reduces a token's score proportionally to how many times it has already appeared. Say "database" five times, and "database" gets pushed down five penalty-units' worth.

Presence penalty reduces a token's score by a flat amount if it has appeared at all - once or fifty times, same penalty.

The practical distinction: frequency penalty fights word-level repetition (the model saying "delve" in every sentence); presence penalty pushes for topic-level novelty (nudging the model to move on to new subjects rather than circling).

When do you need them? Honestly - with modern frontier models, rarely. Repetition loops were a real problem with older and smaller models; GPT-4-class and Claude-class models rarely loop at sensible temperatures. My production defaults are 0 and 0. The two cases where I reach for them:

1. Long-form generation with small/self-hosted models, where repetition loops still genuinely happen. A frequency penalty of 0.3–0.8 usually breaks the loop.
2. List-style brainstorming ("give me 30 marketing angles"), where a presence penalty of 0.5–1.0 measurably reduces near-duplicate items.

And a caution from experience: penalties apply to tokens, not ideas. A high frequency penalty on structured output is quietly destructive - if the model must emit the JSON key "status" twenty times, a strong frequency penalty makes each occurrence progressively less likely, and around item fifteen the model starts misspelling its own schema. I have debugged exactly this. It was not fun, and the fix was one parameter.

Tip: Penalties are for symptoms you can name. "Output repeats the same phrase" → frequency penalty. "List items are near-duplicates" → presence penalty. If you cannot name the symptom, leave them at zero. Parameters set defensively "just in case" are the ones nobody remembers the reason for a year later.

Max Tokens and Stop Sequences: The Brakes

These two do not shape the distribution - they end generation.

max_tokens

A hard ceiling on output length. Generation halts mid-sentence when the ceiling is hit, and you get finish_reason="length" - the truncation state from the previous topic.

Two things engineers get wrong about it:

It is a limit, not a target. Setting max_tokens=2000 does not encourage longer answers; the model stops naturally whenever it stops. Output length is controlled by prompting ("respond in 2–3 sentences"), not by this parameter.

It is your cost and latency circuit breaker. Because output tokens are the expensive tokens and latency scales with output length, max_tokens is your worst-case bound on both. A runaway generation - a model stuck producing a 4,000-token ramble - costs you money and holds a connection open for a minute. Set max_tokens to a generous margin above your expected output (I use roughly 2x the P95 expected length), not to the model's maximum. If your summaries should be ~200 tokens, max_tokens=500 is a circuit breaker; max_tokens=4096 is a blank cheque.

Stop sequences

Strings that terminate generation immediately when the model produces them - the output ends before the stop sequence, which is not included. Up to four sequences on most APIs.

The classic production use: you prompt the model to produce one section of a structured format, and you use the delimiter of the next section as a stop sequence, guaranteeing it never rambles past its assignment.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Answer the question, then on a new line write 'SOURCES:' followed by citations."},
        {"role": "user", "content": question},
    ],
    max_tokens=400,
    stop=["SOURCES:"],   # we only want the answer body this call
)

Stop sequences are cheap, deterministic, and criminally underused. Any time you find yourself writing post-processing code to cut a response off at a known marker, ask whether a stop sequence could have prevented the extra tokens from being generated - and billed - in the first place.

Putting It Together: A Parameter Playbook

Here is the full request with everything we have covered, annotated with my reasoning:

response = client.chat.completions.create(
    model="gpt-4o-mini",        # small model first; escalate with evidence
    messages=messages,
    temperature=0.2,            # extraction task: consistency over variety
    top_p=1.0,                  # one dial at a time; temperature is my dial
    frequency_penalty=0.0,      # no named symptom, so zero
    presence_penalty=0.0,       # same
    max_tokens=500,             # ~2x expected P95 output; circuit breaker
    stop=["\n\n---"],           # our downstream format's section delimiter
)

And the diagnostic playbook - the version of this lesson you will actually use at 2 AM:

Interview: A favourite AI Engineering interview question: "You are building an API that extracts invoice fields into JSON. What temperature do you use and why?" The answer they want is not just "zero" - it is the reasoning: extraction has a correct answer, variance is pure downside, so you sharpen the distribution toward the most probable tokens; and even then you validate the output because temperature 0 is not determinism. Candidates who can explain why through the distribution model stand out immediately from candidates who memorised "low temperature = factual."

Exercise: See the Distribution Move

Exercise: Extend your instrumented client from topic 1.1 into a parameter lab:

1. Pick a prompt with a strongly expected answer ("The capital of France is"). Run it 10 times each at temperature 0, 0.7, and 1.5. Record how many distinct responses you get at each setting.
2. Pick an open-ended prompt ("Suggest a name for a coffee shop"). Repeat the experiment. Compare the variance pattern against the factual prompt - you are watching distribution sharpness interact with temperature.
3. Ask for a 3-item list with max_tokens=30. Confirm the truncation, check finish_reason, then fix it two different ways: raising the limit, and prompting for shorter items. Note which approach changed cost.
4. Challenge: prompt the model to "answer, then explain your reasoning" and use a stop sequence to capture only the answer. Compare completion_tokens with and without the stop sequence - you have just measured stop sequences saving you money.

What You Should Take From This Lesson

One mental model carries everything: the model produces a probability distribution over next tokens, and parameters either reshape it (temperature, penalties), truncate it (top-p, top-k), or halt the loop (max_tokens, stop sequences). "Creativity" settings are really willingness-to-pick-unlikely-tokens settings. Tune one dial at a time. Zero is not determinism. Penalties need a named symptom. max_tokens is a circuit breaker, not a target.

You now control how the model generates. The next topic tackles what it generates - getting reliable, structured, machine-parseable output from a system that fundamentally produces free text. That is where LLM calls stop being demos and start being components you can build on.

Back to AI Engineering