Free lesson · Operating Systems
Thread Pools
The Problem with Creating Threads on Demand
Consider a web server that handles HTTP requests. Every time a client connects, the server needs an execution unit to process the request - read the HTTP headers, run the application logic, query the database, format the response, and send it back.
The simplest approach is to create a new thread for each incoming request. When the request arrives, spawn a thread. When the request is done, destroy the thread. This is the thread-per-request model, and it works fine at low concurrency - say, 50 concurrent requests.
But what happens at 10,000 concurrent requests per second? You are creating and destroying 10,000 threads every second. Each thread creation requires a system call to the kernel. Each one allocates a stack (typically 1 MB). Each one creates a Thread Control Block. And when the request finishes, all of that is torn down - only to be recreated moments later for the next request.
The overhead is enormous. On a server handling 10,000 requests per second with an average request duration of 100 milliseconds, you have 1,000 threads alive at any moment. Each one consumed microseconds of CPU time just for creation and destruction. Each one consumed memory for its stack. And the constant churn of creating and destroying threads forces the OS to do unnecessary work - allocating and freeing memory, updating the process's thread table, and managing scheduler data structures.
The solution is a thread pool: create a fixed number of threads once at startup, reuse them for every request, and never create or destroy threads during normal operation.
How Thread Pools Work
A thread pool is a pattern where you create N worker threads upfront and a shared work queue. The threads sit idle, waiting for work. When a task arrives (like an HTTP request), it is placed on the queue. An idle worker thread picks up the task, processes it, and then goes back to waiting for the next task. The thread is never destroyed - it just loops between "waiting for work" and "doing work."
sequenceDiagram
participant Client as Incoming Requests
participant Queue as Work Queue
participant T1 as Worker Thread 1
participant T2 as Worker Thread 2
participant T3 as Worker Thread 3
Client->>Queue: Request A arrives
Client->>Queue: Request B arrives
Client->>Queue: Request C arrives
T1->>Queue: Pick up Request A
T2->>Queue: Pick up Request B
T3->>Queue: Pick up Request C
Note over T1: Processing Request A...
Note over T2: Processing Request B...
Note over T3: Processing Request C...
T1-->>Queue: Done. Waiting for next task.
Client->>Queue: Request D arrives
T1->>Queue: Pick up Request D
Note over T1: Processing Request D...
T2-->>Queue: Done. Waiting for next task.
T3-->>Queue: Done. Waiting for next task.
This sequence diagram shows how a thread pool works. Three worker threads are ready. When Requests A, B, and C arrive, they go into the work queue. Each worker thread picks up one request and processes it. When Worker Thread 1 finishes Request A, it picks up the next task (Request D) from the queue without being destroyed and recreated. The threads are reused across multiple requests.
The key benefits of this pattern are clear:
No thread creation overhead per request. Threads are created once at startup. The per-request cost is just adding a task to the queue and waking up a waiting thread - both extremely fast operations.
Bounded resource usage. With a fixed pool size of N threads, you know exactly how much memory the threads will use (N times the stack size) and how many context switches the scheduler will need to manage (at most N). This predictability is critical for production systems where you need to guarantee that the server will not run out of memory or become unresponsive under load.
Backpressure under overload. When the server receives more requests than the pool can handle, the work queue fills up. This creates natural backpressure - you can reject requests, return "503 Service Unavailable" errors, or apply other overload protection strategies. Without a pool, thread-per-request would just keep creating threads until the system runs out of memory or the context switch overhead makes it unusable.
Warm caches. Because the same threads are reused, their CPU cache lines stay warm. A thread that just finished handling one HTTP request has cache lines containing the HTTP parsing code, the database driver code, and other frequently accessed data. When it picks up the next request, those cache lines are still valid. With thread-per-request, each new thread starts with a cold cache.
Choosing the Right Pool Size
The most important decision when using a thread pool is how many threads to create. Too few threads and the server cannot handle the load - requests pile up in the queue. Too many threads and you waste memory and increase context switch overhead.
The right size depends on the type of work your threads are doing:
CPU-bound work - work that keeps the CPU busy continuously (computing hashes, sorting data, running algorithms). For CPU-bound work, the optimal pool size is roughly equal to the number of CPU cores. More threads than cores means the extra threads just compete for CPU time, adding context switch overhead without doing additional work. If you have 8 cores, a pool of 8-10 threads is appropriate.
I/O-bound work - work that spends most of its time waiting for I/O (database queries, network calls, disk reads). For I/O-bound work, you want more threads than cores because while one thread is waiting for I/O, another can use the CPU. The optimal size depends on the ratio of waiting time to computing time. A common heuristic is: pool size = number of cores × (1 + wait time / service time). If threads spend 90% of their time waiting for I/O and 10% computing, a pool of 8 × (1 + 9) = 80 threads makes sense on an 8-core machine.
Mixed work - most real-world applications. The heuristic above is a starting point, but in practice you should measure and tune. Start with a pool size, run load tests, and adjust based on throughput, latency, and resource usage.
Little's Law and Thread Pool Sizing
There is a mathematical foundation for pool sizing called Little's Law: L = λ × W, where L is the average number of concurrent tasks in the system, λ is the arrival rate (tasks per second), and W is the average time each task takes to complete.
For example, if your server receives 500 requests per second and each request takes an average of 200 milliseconds to process, the average number of concurrent requests is 500 × 0.2 = 100. You need at least 100 threads to handle this load without queueing. In practice, you would set the pool slightly larger (say, 120-150) to handle bursts and variance in request processing time.
Little's Law gives you a principled starting point, but real workloads are not perfectly uniform. Some requests take 10 milliseconds and some take 2 seconds. Request rates vary throughout the day. So the math gives you a floor, and load testing tells you the actual number.
Dynamic Thread Pools
Some thread pool implementations can adjust their size dynamically. Java's ThreadPoolExecutor accepts both a core pool size and a maximum pool size. Under normal load, it maintains the core number of threads. Under heavy load, it creates additional threads up to the maximum. When the load decreases, excess threads are terminated after an idle timeout.
This dynamic approach balances resource efficiency (fewer threads during quiet periods) with responsiveness (more threads during busy periods). However, it adds complexity - the pool must decide when to create and destroy threads, and the creation delay during load spikes can cause latency spikes for the requests that triggered the new thread creation. For systems that need predictable performance, a fixed-size pool is often preferred.
Interview: "How would you size a thread pool?"
"For CPU-bound work, the pool should be roughly equal to the number of CPU cores - more threads just add context switch overhead. For I/O-bound work, I would use more threads than cores because threads spend time waiting for I/O while other threads can use the CPU. A starting point is cores × (1 + wait/compute ratio). In practice, I would start with this estimate, run load tests, and tune based on measured throughput and latency. The key signal to watch is CPU utilization - if it is well below 100% and latency is high, you probably need more threads. If context switch rates are high and throughput has plateaued, you probably have too many."
Thread Pools in Real Systems
Thread pools are everywhere in production systems. Here are the most common examples you might reference in interviews:
Java's ExecutorService. Java provides a built-in thread pool abstraction. Executors.newFixedThreadPool(200) creates a pool of 200 threads. Tasks submitted to the executor are placed on an internal queue and processed by worker threads. Spring Boot's embedded Tomcat server uses a thread pool of 200 threads by default to handle HTTP requests.
// Java thread pool example
ExecutorService pool = Executors.newFixedThreadPool(200);
// Submit tasks to the pool
pool.submit(() -> {
// Handle HTTP request
String response = processRequest(request);
sendResponse(response);
});
// The thread that processed the request goes back to the pool
// and waits for the next task - no thread creation/destruction
Node.js worker threads. Node.js uses a single-threaded event loop for I/O, but it has a thread pool (via libuv) for operations that cannot be done asynchronously - file system operations, DNS lookups, and CPU-intensive work. The default pool size is 4 threads, which you can increase with the UV_THREADPOOL_SIZE environment variable.
Database connection pools. While not exactly thread pools, connection pools follow the same pattern. A database connection pool creates N connections to the database at startup. When a thread needs to query the database, it borrows a connection from the pool, uses it, and returns it. This avoids the overhead of creating a new TCP connection and authenticating with the database for every query. HikariCP (Java's most popular connection pool) defaults to a pool size of 10 connections.
Go's goroutine scheduler. Go's runtime is itself a thread pool. It creates a small number of OS threads (one per CPU core by default) and multiplexes thousands or millions of goroutines across them. Each OS thread picks goroutines from a run queue and executes them. When a goroutine blocks on I/O, the OS thread picks up another goroutine instead of sitting idle. This is the many-to-many threading model implemented as a built-in thread pool.
graph TD
subgraph TPR["Thread-Per-Request Model"]
R1["Request 1"] --> T1C["Create Thread 1"]
R2["Request 2"] --> T2C["Create Thread 2"]
R3["Request 3"] --> T3C["Create Thread 3"]
T1C --> T1D["Destroy Thread 1"]
T2C --> T2D["Destroy Thread 2"]
T3C --> T3D["Destroy Thread 3"]
end
subgraph TP["Thread Pool Model"]
R4["Request 1"] --> Q["Work Queue"]
R5["Request 2"] --> Q
R6["Request 3"] --> Q
Q --> W1["Worker 1<br/>(reused)"]
Q --> W2["Worker 2<br/>(reused)"]
Q --> W3["Worker 3<br/>(reused)"]
end
This diagram compares the thread-per-request model with the thread pool model. In thread-per-request, each request triggers thread creation and destruction - three requests mean three create-and-destroy cycles. In the thread pool model, requests go into a work queue, and three persistent worker threads pick them up. The workers are reused for every subsequent request, avoiding the creation and destruction overhead.
Queue Strategies and Overload Handling
The work queue in a thread pool is not just a passive buffer - it is a critical component that affects system behavior under load. There are several queue strategies worth understanding for interviews:
Unbounded queue. The queue can grow without limit. This is simple but dangerous - under heavy load, the queue can grow until the server runs out of memory. It also increases latency indefinitely - a request that enters a queue with 10,000 items ahead of it will wait a very long time.
Bounded queue. The queue has a fixed maximum size. When the queue is full, new tasks are rejected. This provides overload protection - the server fails gracefully by returning errors to excess requests rather than accepting more work than it can handle and slowing down for everyone.
Priority queue. Tasks have priorities, and higher-priority tasks are processed before lower-priority ones. This is useful when some requests are more important than others - for example, health check requests should be processed quickly even under heavy load.
Work-stealing queue. Each worker thread has its own queue. When a thread finishes its tasks and its queue is empty, it "steals" tasks from another thread's queue. This provides good load balancing without a single contended shared queue. Java's ForkJoinPool uses this pattern, and Go's goroutine scheduler uses a similar approach.
What Happens When the Pool Is Exhausted
The behavior when all pool threads are busy and the queue is full is called the rejection policy, and it is one of the most important aspects of a thread pool for production systems. Different policies are appropriate for different situations:
Abort. Throw an exception or return an error immediately. The caller knows the server is overloaded and can retry later or route the request elsewhere. This is the most common policy for web servers - return a 503 status code and let the load balancer route to another server.
Caller-runs. The thread that submitted the task (typically the thread that accepted the network connection) runs the task itself. This provides a natural form of throttling - the accepting thread cannot accept new connections while it is running a task, which slows down the rate of incoming requests. This is a self-regulating mechanism that prevents the queue from growing unboundedly.
Discard. Silently drop the task. This is appropriate for tasks where occasional loss is acceptable - like updating a non-critical counter or sending a best-effort notification.
Discard-oldest. Drop the oldest task in the queue (the one that has been waiting the longest) and add the new task. This is appropriate when fresher requests are more valuable than stale ones - like real-time data processing where old data is already outdated.
Understanding these rejection policies and being able to discuss which one is appropriate for a given system is a strong signal in system design interviews.
Tip: For system design interviews: Mentioning bounded queues and backpressure shows the interviewer that you think about failure modes. A server that accepts unlimited requests and degrades gracefully is better than one that accepts unlimited requests and crashes. Always mention what happens when the pool is fully busy and the queue is full.
Exercise: Starter: In your primary programming language, create a simple thread pool with 4 worker threads and a shared queue. Submit 20 tasks that each sleep for 1 second and print which thread handled them. Observe that all 20 tasks complete in about 5 seconds (20 tasks ÷ 4 threads = 5 rounds).
Intermediate: Modify the thread pool from the Starter exercise to use a bounded queue with a maximum size of 5. Submit 20 tasks rapidly. What happens when the queue is full? How would you handle the rejected tasks in a production system?
Advanced: Benchmark the thread-per-request model against the thread pool model. Create a test that submits 10,000 short tasks (each doing a small computation). In the first version, create a new thread for each task. In the second version, use a thread pool of 100 threads. Measure the total time for both approaches. How much faster is the thread pool? What is the speedup ratio?