Free lesson · AI Engineering

Making Your First LLM API Call

Easy12 min readaillmrag

Making Your First LLM API Call

The Most Expensive HTTP Request You Will Ever Make

Let me start with a story that sets the stakes.

In my second month at Meta, a product team shipped a feature that summarised user feedback threads. Simple idea, clean implementation, worked beautifully in testing. Three weeks after launch, the team got a message from finance asking why their infrastructure budget line had grown by $40,000 in a month.

The bug was not in the code. The code was correct. The bug was in the mental model. The engineers treated the LLM call like any other HTTP request - like calling a weather API or fetching a user profile. Fire the request, get the response, move on. But an LLM call is not like other HTTP requests in one critical way: the cost scales with the size of what you send and what you get back. They were sending entire feedback threads - sometimes 30,000 tokens - when a 2,000-token extract would have produced the same summary.

Nobody had done anything wrong by traditional engineering standards. The request succeeded. The response was correct. The latency was acceptable. Every metric a backend engineer normally watches looked green. The metric that mattered - cost per request - was not on anyone's dashboard.

I open with this because your first LLM API call is going to look deceptively familiar. It is an HTTP POST with a JSON body. You have made ten thousand of these. But underneath that familiar shape is a fundamentally different kind of service - non-deterministic, token-metered, variable-latency - and the habits you build in this first lesson determine whether you become the engineer who ships reliable AI features or the engineer who gets the message from finance.

So we are going to make your first call, and we are going to make it the right way: understanding every field in the request, every field in the response, and where the money goes.

The Anatomy of an LLM API Request

Every major LLM provider - OpenAI, Anthropic, Google - exposes their models through a chat-completion-style API. The shapes differ slightly between providers, but the structure is the same everywhere:

1. A model identifier - which model you want
2. A list of messages - the conversation so far
3. Generation parameters - how the model should generate (covered in depth in the next topic)

Here is the minimal request to OpenAI:

# Note: this example shows the structure. To actually run it,
# you need OPENAI_API_KEY set in your environment.
import os

request_body = {
    "model": "gpt-4o-mini",
    "messages": [
        {"role": "user", "content": "Explain what a token is in one sentence."}
    ],
    "max_tokens": 100
}

# This is literally all an LLM API call is: an HTTP POST with this JSON.
import json
print(json.dumps(request_body, indent=2))

And here is the same call using the official SDK, which is how you will write it in practice:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": "Explain what a token is in one sentence."}
    ],
    max_tokens=100,
)

print(response.choices[0].message.content)

Let me walk through each piece, because every field carries a decision.

The Model Identifier

"gpt-4o-mini" is a model string. Every provider has a menu of models at different capability and price points. Choosing the model is the single biggest cost decision you make - the price difference between a provider's flagship model and their small model is typically 20–60x per token.

For learning and for most structured tasks (extraction, classification, reformatting), start with the small models: gpt-4o-mini, claude-haiku-4-5, or a local Llama via Ollama. Escalate to flagship models only when the task demonstrably needs them. We formalise this decision in the model routing module; for now, the habit to build is: small model first, prove you need bigger.

The Messages Array

The messages array is the conversation. Each message has a role and content. Three roles matter:

The system message is where production applications live or die. It is your control surface - the place you define the model's job, its constraints, its output format, and its boundaries. A production system message is often 500–2,000 tokens of carefully engineered instruction. We dedicate an entire module to this. For now, know that it exists and that it is the first thing you reach for when the model misbehaves.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a technical glossary. Define terms in exactly one sentence, no preamble."
        },
        {
            "role": "user",
            "content": "token"
        }
    ],
    max_tokens=60,
)

Run those two versions side by side and you will see the difference immediately: without the system message you get a friendly paragraph; with it you get one clean sentence. Same model, same user input, different behaviour. That is the system message doing its job.

Important: The API is stateless. The model does not remember your previous calls. Every request must contain the entire conversation history in the messages array. If your chat application has a 20-turn conversation, all 20 turns go into every request - and you pay for all of them, every time. This single fact drives an enormous amount of AI application architecture, and we handle it properly in the conversation management topic.

The Anatomy of the Response

The response is where most tutorials wave their hands, and it is exactly where production engineers need to slow down. Here is what comes back:

{
  "id": "chatcmpl-abc123",
  "model": "gpt-4o-mini-2024-07-18",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A token is a chunk of text - roughly three-quarters of a word - that language models use as their basic unit of reading and writing."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 27,
    "completion_tokens": 31,
    "total_tokens": 58
  }
}

Three fields deserve your full attention.

choices[0].message.content

The generated text. This is what tutorials show you and stop. Notice it lives inside an array (choices) - you can request multiple completions per call, though in production you almost never do.

finish_reason

The field almost everyone ignores and absolutely should not. It tells you why the model stopped generating:

length is the one that bites people. If you set max_tokens=100 and the model needed 150, you get 100 tokens of a sentence that just... stops. If your code parses that output - say, as JSON - the parse fails, and now you have a production incident caused not by the model but by ignoring a field the API handed you for free.

choice = response.choices[0]
if choice.finish_reason == "length":
    # Truncated output. Do NOT parse it as if it were complete.
    # Retry with a higher max_tokens, or log and degrade gracefully.
    raise TruncatedResponseError(usage=response.usage)

Check finish_reason on every call. Make it a habit from call number one.

usage

The bill. prompt_tokens is what you sent, completion_tokens is what came back, and you pay different rates for each - output tokens typically cost 3–4x input tokens. Let me make this concrete with real arithmetic, because vague cost intuitions are how you get the $40,000 surprise:

Example: Cost of the call above at gpt-4o-mini pricing (~$0.15 per million input tokens, ~$0.60 per million output tokens):

Component	Tokens	Rate	Cost
Input	27	$0.15/M	$0.00000405
Output	31	$0.60/M	$0.0000186
Total	58	-	~$0.000023

Effectively free at one call. Now scale it: same structure, but a RAG application sending 8,000-token prompts and getting 500-token answers, at 50,000 requests/day, on a flagship model at ~$3/M input and ~

5/M output - that is

,200 input + $375 output = ~

,575 per day, or ~$47,000 per month. Same code shape. Four orders of magnitude difference in cost. The usage field is how you see it coming.

From your very first application, log usage on every call. Not as an optimisation - as basic observability, the same way you log status codes.

Calling Anthropic: Same Shape, Instructive Differences

Here is the equivalent call to Claude:

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=100,
    system="You are a technical glossary. Define terms in exactly one sentence.",
    messages=[
        {"role": "user", "content": "token"}
    ],
)

print(response.content[0].text)
print(response.stop_reason)   # Anthropic's name for finish_reason
print(response.usage)          # input_tokens / output_tokens

Three differences worth noticing, because they teach you something about the API landscape:

The system prompt is a top-level parameter, not a message in the array. Different surface, same concept.

max_tokens is required, not optional. Anthropic forces you to make the truncation decision explicitly rather than defaulting it. Frankly, I think this is the better design - it makes the failure mode visible on day one.

The response content is a list of blocks, not a single string, because Claude responses can contain mixed content types (text, tool calls). You take response.content[0].text for simple text.

The lesson underneath the differences: do not memorise provider APIs - learn the invariant structure. Model, messages, parameters in; content, stop reason, usage out. Every provider is a dialect of that grammar. When a new provider or a new API version ships, you will orient in minutes because you know what to look for.

Tip: Write a thin wrapper from day one. Even a 30-line function that takes (system, user_message, model) and returns (text, finish_reason, usage) - and logs the usage - will save you from scattering provider-specific code through your application. When you want to switch providers, or add a second one, you change one file. Every production AI codebase I have worked in eventually builds this layer. Build it on day one and it costs you nothing.

What Actually Happens When You Send a Prompt

You do not need to understand transformer internals yet - that is Module 2 - but you need a correct high-level model of the round trip, because it explains the two properties of LLM calls that will shape everything you build: variable latency and non-determinism.

sequenceDiagram
    participant App as Your Application
    participant API as Provider API
    participant Model as Model (GPU cluster)
    App->>API: POST /chat/completions (prompt)
    API->>API: Auth, rate limit check, safety screening
    API->>Model: Tokenise prompt, load into context
    Model->>Model: Generate output one token at a time
    Model->>API: Complete token sequence
    API->>App: JSON response (content + usage + finish_reason)

This sequence diagram shows the round trip of an LLM API call. Your application sends a POST request to the provider API, which performs authentication, rate limiting, and safety screening. The API tokenises the prompt and sends it to the model running on a GPU cluster. The model generates output one token at a time, returns the complete sequence to the API, and the API returns a JSON response containing the content, usage, and finish reason to your application.

The key phrase is one token at a time. The model does not compose its answer and then send it. It generates the first token, then generates the second token conditioned on everything so far, and so on until it produces a stop signal or hits your limit. Two consequences fall directly out of this:

Latency scales with output length. A 50-token answer might return in 800ms; a 1,500-token answer from the same model might take 20 seconds. Input size affects latency far less than output size does. When an AI feature feels slow, the first question is always: how many tokens is it generating, and does it need to generate that many?

The same prompt can produce different outputs. At each token, the model produces a probability distribution over possible next tokens and samples from it. Sampling is random by default. Run the identical request three times and you may get three differently-worded responses. This is not a bug and it is not fixable with a config flag alone - it is the fundamental property that separates AI Engineering from everything you have built before, and the next topic is entirely about the parameters that control it.

Interview: This comes up in AI Engineering interviews constantly, usually disguised as a system design question: "Your LLM-powered endpoint has a P99 latency of 25 seconds. Walk me through your diagnosis." Strong candidates immediately reach for output token count, model size, and streaming - because they understand generation is sequential and output-bound. Weak candidates talk about database indexes and CDNs. The mental model from this lesson is the difference.

Error Handling: The Part Tutorials Skip

LLM APIs fail in ways you must plan for from the first line of production code. The four failures you will actually see:

429 - Rate limited. You exceeded your requests-per-minute or tokens-per-minute quota. This is not exceptional; at any real traffic level it is routine. Handle with exponential backoff and respect the retry-after header.

5xx - Provider outage or overload. The major providers have had multiple public incidents per year, every year. Your application needs an answer to "what do we do when the model is down" - a fallback model, a queue, or graceful degradation. "We are down too" is a choice, but it should be a deliberate one.

Timeouts. Because latency is output-bound and variable, a request that normally takes 3 seconds occasionally takes 60. Set explicit client timeouts. The default in many HTTP libraries is either infinite or far too long for a user-facing request path.

Content filter interventions. The provider's safety layer can block a request or truncate a response. Your code must treat an empty or filtered response as a handleable state, not crash on a missing field.

import time
from openai import OpenAI, RateLimitError, APIStatusError, APITimeoutError

client = OpenAI(timeout=30.0)  # explicit timeout, always

def call_llm(messages, model="gpt-4o-mini", max_retries=3):
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model, messages=messages, max_tokens=500
            )
            choice = response.choices[0]
            if choice.finish_reason == "length":
                # truncated - decide policy: retry bigger, or flag
                pass
            return choice.message.content, response.usage
        except RateLimitError:
            wait = 2 ** attempt  # 1s, 2s, 4s
            time.sleep(wait)
        except APITimeoutError:
            if attempt == max_retries - 1:
                raise
    raise RuntimeError("LLM call failed after retries")

This is not the final production pattern - retries interact with cost, idempotency, and user experience in ways we cover in the reliability module. But retry-with-backoff, explicit timeouts, and finish-reason checks are the floor, and the floor starts now.

Warning: Retries cost money. A failed traditional API call costs you nothing but time. A timed-out LLM call may have already generated (and billed you for) most of a response before your client gave up. Blind aggressive retries on an LLM endpoint can double or triple your token spend during a provider slowdown - exactly the moment costs spike anyway. Log every retry. You will thank yourself.

Exercise: Your First Instrumented Call

Exercise: Build a minimal instrumented LLM client. Write a script that:

1. Sends the same prompt to a small model five times and prints all five responses. Observe the variation - this is non-determinism made visible.
2. Logs, for every call: model, prompt_tokens, completion_tokens, latency in ms, and finish_reason.
3. Computes and prints the total cost of the five calls using your provider's current per-token pricing.
4. Challenge: set max_tokens=20 and ask a question that needs a long answer. Confirm you get finish_reason="length" and write the handling branch for it.

Keep this script. We extend it in every topic of this module, and by topic 1.5 it becomes the skeleton of your first real project.

What You Should Take From This Lesson

An LLM API call is an HTTP POST that looks familiar and behaves like nothing you have used before. It is metered by token, priced asymmetrically between input and output, stateless across calls, sequential in generation, variable in latency, and non-deterministic in output. Every one of those properties showed up in this single lesson because every one of them shows up in every AI application you will ever build.

The habits to carry forward: check finish_reason always, log usage always, set explicit timeouts always, and reach for the small model first.

Next, we open up the parameters - temperature, top-p, penalties, stop sequences - and I will show you exactly what each one does to that probability distribution the model samples from, including the settings I use in production and the ones I have learned to never touch.

Back to AI Engineering