Free lesson · Python Basic to Advance

HTTP Clients with `requests`

Hard10 min readpython

HTTP Clients with `requests`

What `requests` Is

requests is the most widely used third-party HTTP client library for Python. It provides a clean, high-level API for making HTTP requests - GET, POST, PUT, DELETE, and more - with sensible defaults, automatic decoding, session management, and a much nicer interface than the standard library's urllib.

import requests

response = requests.get("https://api.example.com/users")
print(response.status_code)    # 200
print(response.json())         # parsed JSON
print(response.headers)        # response headers

This code uses requests for a basic GET. The library handles URL parsing, connection setup, response decoding, and JSON parsing. The result is a Response object exposing status code, headers, and body. Compared to urllib - Topic 26.2 - requests is dramatically more ergonomic for typical use.

Install with pip install requests. It's not in the standard library but is one of the most common third-party packages - installed on nearly every Python project that talks to HTTP.

The Standard HTTP Verbs

The library exposes one function per HTTP method:

import requests

# GET - fetch a resource
r = requests.get("https://api.example.com/users")

# POST - create a resource
r = requests.post("https://api.example.com/users", json={"name": "alice"})

# PUT - replace a resource
r = requests.put("https://api.example.com/users/1", json={"name": "alice"})

# PATCH - modify part of a resource
r = requests.patch("https://api.example.com/users/1", json={"email": "new@example.com"})

# DELETE - remove a resource
r = requests.delete("https://api.example.com/users/1")

# HEAD - fetch headers only
r = requests.head("https://api.example.com/users/1")

This code shows the verb functions. Each maps to the corresponding HTTP method. The url is the first argument; payload, headers, and other options are keyword arguments. The semantics - GET for reading, POST for creating, and so on - are conventions of the API being called, not enforced by requests itself.

Query Parameters

For URL query strings, pass a dict via params:

import requests

# requests builds the URL: https://api.example.com/search?q=python&limit=10
r = requests.get(
    "https://api.example.com/search",
    params={"q": "python", "limit": 10},
)

This code uses params. The library URL-encodes values, joins them with ampersands, and appends to the URL. Useful - never need to manually escape special characters or worry about leading question marks. Lists become repeated parameters: passing a list means several values for the same key.

For lists of values, the same key appears multiple times in the URL:

# Becomes ?tag=python&tag=web
r = requests.get("...", params={"tag": ["python", "web"]})

This code passes a list as a query parameter. Each list element produces one occurrence of the key in the final URL - the convention APIs use for multi-value filters.

Request Bodies

For sending data in the body, several options:

import requests

# JSON - most common for APIs
r = requests.post(
    "https://api.example.com/users",
    json={"name": "alice", "email": "alice@example.com"},
)
# Automatically sets Content-Type: application/json and serializes

# Form-encoded - like HTML form POST
r = requests.post(
    "https://example.com/login",
    data={"username": "alice", "password": "secret"},
)
# Sets Content-Type: application/x-www-form-urlencoded

# Raw bytes
r = requests.post(
    "https://example.com/upload",
    data=b"some binary content",
    headers={"Content-Type": "application/octet-stream"},
)

# Multipart file upload
with open("file.pdf", "rb") as f:
    r = requests.post(
        "https://example.com/upload",
        files={"file": ("report.pdf", f, "application/pdf")},
    )

This code shows body types. JSON parameter accepts a dict and handles serialization plus content-type. Data parameter accepts a dict for form-encoded or bytes for raw. Files parameter handles multipart uploads - the canonical way to send file content. Choose based on what the API expects.

For modern REST APIs, json=... is the most common form. For HTML form submissions or legacy APIs, data=....

Reading Responses

The Response object provides several ways to access the body:

import requests

r = requests.get("https://api.example.com/users/1")

# Status code
print(r.status_code)             # 200
print(r.ok)                       # True for 2xx
print(r.reason)                   # "OK"

# Headers
print(r.headers["Content-Type"])  # application/json
print(r.headers.get("X-RateLimit-Remaining"))

# Body
print(r.text)         # decoded string
print(r.content)      # raw bytes
print(r.json())       # parsed JSON

This code shows response access. Status code as an int. OK is a convenience for 2xx range. Text decodes using the response's charset; content is the raw bytes. JSON parses and returns the structure. Choose based on what the response actually is - text for HTML, content for binary, json for API responses.

For unknown content types, check r.headers["Content-Type"] before assuming the format.

Headers

Custom headers are passed via the headers parameter:

import requests

r = requests.get(
    "https://api.example.com/users",
    headers={
        "Authorization": "Bearer abc123",
        "User-Agent": "MyApp/1.0",
        "Accept": "application/json",
    },
)

This code sets request headers. The dictionary maps header names to values. Common headers: Authorization for credentials, User-Agent identifying the client, Accept for content negotiation. Topic 26.7 covers authentication patterns including bearer tokens and API keys.

Timeouts

Always specify a timeout. Without one, requests waits forever for a response:

import requests

# Single timeout - for everything
r = requests.get("https://api.example.com/users", timeout=10)

# Separate connect and read timeouts (tuple)
r = requests.get("https://api.example.com/users", timeout=(5, 30))
# 5 seconds to connect, 30 seconds to read the response

# No timeout - bad! Can hang forever
r = requests.get("https://api.example.com/users")    # never do this in production

This code shows timeouts. A single value applies to both connection and read. A tuple lets you specify them separately - useful when slow servers need more read time but fast connections are expected. The default is no timeout, which means a stuck server hangs the calling code forever. Production code should always specify a timeout.

The most common production bug with requests is a missing timeout - a hung remote service freezes the caller indefinitely.

Error Handling

requests doesn't raise on HTTP errors (4xx, 5xx) by default. Check status_code or use raise_for_status:

import requests

# Manual check
r = requests.get("https://api.example.com/users/999")
if r.status_code == 404:
    print("not found")
elif r.ok:
    print(r.json())

# raise_for_status - raises for 4xx/5xx
try:
    r = requests.get("https://api.example.com/users/999", timeout=10)
    r.raise_for_status()
    data = r.json()
except requests.HTTPError as e:
    print(f"HTTP error: {e.response.status_code}")
except requests.Timeout:
    print("timeout")
except requests.ConnectionError:
    print("connection failed")
except requests.RequestException as e:
    print(f"request failed: {e}")

This code shows error handling. By default, a 404 or 500 response returns normally - the response has the status code; no exception is raised. Raise for status converts non-2xx responses to HTTPError exceptions. Connection failures, DNS errors, and timeouts have their own exception classes. Catching RequestException covers all of them.

The exception hierarchy:

RequestException - base class
- HTTPError - non-2xx response (after raise_for_status)
- Timeout - connection or read timeout
- ConnectionError - network failure
- TooManyRedirects - redirect loop
- URLRequired - missing URL

Sessions

For repeated requests to the same host, a Session reuses the underlying connection (HTTP keep-alive), persists cookies, and lets you set defaults:

import requests

with requests.Session() as session:
    # Set defaults that apply to all requests
    session.headers["Authorization"] = "Bearer abc123"
    session.headers["User-Agent"] = "MyApp/1.0"

    # Connection reuse - much faster than separate requests
    for user_id in range(1, 100):
        r = session.get(f"https://api.example.com/users/{user_id}")

This code uses a session. Connection pooling reuses TCP connections - much faster than opening a new connection per call. Headers set on the session apply to all requests made through it. Useful for making many calls to the same host - scraping, API clients, batch processing. The with block ensures the connections are closed.

For more than a handful of requests to the same host, sessions are essential - connection setup dominates per-request time otherwise.

Authentication

requests supports several authentication schemes via the auth parameter:

import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

# Basic auth
r = requests.get("https://api.example.com/", auth=("user", "password"))
# or explicit
r = requests.get("https://api.example.com/", auth=HTTPBasicAuth("user", "password"))

# Digest auth
r = requests.get("https://api.example.com/", auth=HTTPDigestAuth("user", "password"))

# Bearer token - manual header
r = requests.get(
    "https://api.example.com/",
    headers={"Authorization": "Bearer abc123"},
)

# API key in header
r = requests.get(
    "https://api.example.com/",
    headers={"X-API-Key": "abc123"},
)

This code shows auth schemes. Tuple form for basic auth is the shortest. HTTPBasicAuth is more explicit. Bearer tokens and API keys are header-based - set them via headers. Topic 26.7 covers authentication patterns more broadly - what to send, when, and how to handle expiration.

Following Redirects

By default, requests follows redirects up to 30 hops:

import requests

# Follows redirects
r = requests.get("https://example.com")    # may end at https://www.example.com

# Don't follow
r = requests.get("https://example.com", allow_redirects=False)
print(r.status_code)    # 301 or 302
print(r.headers["Location"])

# Inspect the redirect chain
r = requests.get("https://example.com")
for past in r.history:
    print(past.status_code, past.url)
print(r.url)    # final URL

This code controls redirects. Default behavior follows them - useful for most code. Disable when you need to inspect the redirect itself - for example, checking what an authentication flow redirects to. History shows the chain of intermediate responses; the final response is the result.

Streaming Large Responses

For large responses, streaming avoids loading everything into memory:

import requests

# Default - full body loaded
r = requests.get("https://example.com/large-file")
print(len(r.content))

# Streaming - read in chunks
with requests.get("https://example.com/large-file", stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("output.dat", "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

This code streams a response. Stream true delays loading the body until iterated. Iter content reads in chunks of the specified size. Useful for downloading large files, processing data line by line, or any case where the full response would exceed memory. The with statement ensures the connection is released when done.

For files larger than a few megabytes, streaming is essential.

Retries

requests doesn't retry by default. For automatic retries with backoff, use urllib3.util.Retry with a HTTPAdapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()

retry = Retry(
    total=5,
    backoff_factor=0.5,
    status_forcelist=[500, 502, 503, 504],
    allowed_methods=["GET", "POST"],
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)

# Now session.get auto-retries on listed status codes and connection errors
r = session.get("https://api.example.com/flaky", timeout=10)

This code adds retries. Retry policy: up to five attempts, exponential backoff via the backoff factor, retry on specific HTTP status codes, allow retries on these methods. Mount applies the adapter to all requests through the session. Useful for production code calling external APIs - transient failures (5xx, network blips) usually succeed on retry. Topic 25.5 covered idempotency, which matters here - retried POSTs should be idempotent.

For idempotent operations (GET, PUT, DELETE), retries are usually safe. For POST, consider whether the API supports idempotency keys.

A Complete Example

A realistic API client:

import logging
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

logger = logging.getLogger(__name__)

class APIClient:
    def __init__(self, base_url, token, timeout=10):
        self.base_url = base_url.rstrip("/")
        self.timeout = timeout

        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {token}",
            "Accept": "application/json",
        })

        retry = Retry(
            total=3,
            backoff_factor=0.5,
            status_forcelist=[500, 502, 503, 504],
            allowed_methods=["GET", "PUT", "DELETE"],
        )
        adapter = HTTPAdapter(max_retries=retry)
        self.session.mount("https://", adapter)

    def get_user(self, user_id):
        url = f"{self.base_url}/users/{user_id}"
        logger.debug("fetching user %s", user_id)
        try:
            r = self.session.get(url, timeout=self.timeout)
            r.raise_for_status()
            return r.json()
        except requests.HTTPError:
            if r.status_code == 404:
                return None
            raise
        except requests.RequestException:
            logger.exception("request failed for user %s", user_id)
            raise

    def __enter__(self):
        return self

    def __exit__(self, *args):
        self.session.close()

This code shows a realistic client. The session holds defaults - authorization, accept header, retry policy. Each method composes the URL, makes the call, raises on errors except 404 which returns None. Context manager protocol lets callers use with blocks. Logging records the operation; exception logging captures failures with traceback. This is the shape most production HTTP-client code takes.

This is the pattern most production HTTP-client code follows - class with a session, retry config, sensible timeouts, structured error handling.

Cookies

requests handles cookies automatically within a session:

import requests

with requests.Session() as session:
    # Login sets a cookie
    session.post("https://example.com/login", data={"user": "alice", "password": "secret"})

    # Subsequent requests include the cookie automatically
    r = session.get("https://example.com/dashboard")

# Explicit cookies
r = requests.get("https://example.com/", cookies={"session_id": "abc"})

# Inspect cookies
print(session.cookies.get_dict())

This code handles cookies. A session collects them automatically - login sets them, subsequent requests send them back. Explicit cookies via the cookies parameter. The session's cookies attribute is a CookieJar that can be inspected or modified directly.

Async Alternative

For async code, requests blocks the event loop. httpx (pip install httpx) provides a requests-compatible API with async support:

import httpx
import asyncio

# Sync - drop-in for requests
r = httpx.get("https://api.example.com/users")

# Async
async def fetch():
    async with httpx.AsyncClient(timeout=10) as client:
        r = await client.get("https://api.example.com/users")
        return r.json()

result = asyncio.run(fetch())

This code shows httpx. The sync API closely matches requests. The async API uses async with and await - works inside asyncio code without blocking the event loop. For async services, httpx replaces requests directly. Topic 22.10 covered asyncio basics.

For async services, httpx is the modern choice. For sync code, requests remains the most common.

Performance

requests overhead:

One request without session: ~10–100ms (connection setup dominates)
Session-based: same with connection reuse (much faster after first call)
JSON parsing: microseconds for typical payloads
Large body download: bounded by network bandwidth

For high-throughput needs, sessions plus connection pooling matter. For very high concurrency, async with httpx scales better.

Interview Relevance

Interview: requests is foundational Python:

The verb methods (get, post, etc.)

Query parameters via params, body via json/data/files

Response access: status_code, headers, text, content, json()

Always set a timeout - most common production bug otherwise

Sessions for connection reuse and persistent state

raise_for_status for error handling

The exception hierarchy

Retries via urllib3.util.Retry

httpx as the modern async alternative

A common interview moment: a candidate forgets the timeout. The right framing is that without it, a hung server hangs the caller indefinitely.

Another moment: many requests to the same host, no session. The right fix is connection pooling via a session.

Module 30 covers Python interview preparation.

Common Pitfalls

Warning: No timeout. Hangs forever on slow or dead servers. Always set one.

Treating 4xx as exceptions automatically. Default behavior doesn't raise. Use raise_for_status if that's what you want.

One request per call without a session. Connection setup dominates. Use Session for repeated calls.

Catching Exception broadly. Mask specific failure modes. Catch RequestException or its subclasses.

json= confused with data=. JSON parameter handles serialization and content type; data is for form-encoded.

No retries for transient failures. External APIs are not 100% reliable. Add retry policy.

stream=True without iterating. Connection stays open and memory leaks.

Logging full URLs with secrets in query strings. API keys end up in logs. Use headers for credentials.

Try It

Exercise: Practice with requests:

Make a GET request to a public API. Print the JSON body.

POST a JSON body to a test endpoint (e.g., httpbin.org/post). Verify the echo.

Always specify a timeout. Trigger a timeout against a slow endpoint.

Create a Session. Make 10 requests through it. Compare timing against 10 individual requests.get calls.

Add a Retry adapter that retries on 5xx responses.

Stream a large file response and write it to disk in chunks.

Back to Python Basic to Advance

HTTP Clients with `requests`

HTTP Clients with requests

What requests Is

The Standard HTTP Verbs

Query Parameters

Request Bodies

Reading Responses

Headers

Timeouts

Error Handling

Sessions

Authentication

Following Redirects

Streaming Large Responses

Retries

A Complete Example

Cookies

Async Alternative

Performance

Interview Relevance

Common Pitfalls

Try It

HTTP Clients with `requests`

What `requests` Is