Free lesson · Operating Systems

Register a handler for SIGTERM

Easy10 min readoperating-systemsosconcurrency

A Tap on the Shoulder

Every IPC mechanism you have learned so far - pipes, message queues, shared memory, sockets - transfers data between processes. Signals are different. A signal is not a data channel. It is a notification - a lightweight, asynchronous message from the OS or from another process that tells a process "something happened." Think of it like a tap on the shoulder: it interrupts whatever the process is doing and demands attention.

When you press Ctrl+C in a terminal, you are sending a signal (SIGINT) to the foreground process. When the OS detects that a process accessed invalid memory, it sends a signal (SIGSEGV). When a child process terminates, the OS sends a signal (SIGCHLD) to the parent. When you type kill 1234 in a terminal, you are sending a signal (SIGTERM) to process 1234.

Signals are the oldest IPC mechanism in Unix - they predate pipes, sockets, and shared memory. They are also the most misunderstood. Many engineers use signals without understanding how they work, which leads to bugs that are notoriously difficult to reproduce and fix. Understanding signals at the OS level is important for interviews (they come up in questions about process lifecycle, graceful shutdown, and error handling) and for production debugging (signal-related bugs cause crashes, resource leaks, and zombie processes).

How Signals Work

When a signal is sent to a process, the OS does the following:

Step 1: Delivery. The OS sets a flag in the target process's PCB indicating that a signal is pending. The signal is not delivered immediately - it is marked as pending and will be handled the next time the process runs (or immediately, if the process is currently running).

Step 2: Handling. When the process is scheduled to run (or on return from a system call), the kernel checks for pending signals. If a signal is pending, the kernel interrupts the process's normal execution and invokes the signal handler.

Step 3: Signal handler execution. The signal handler is a function that the process registered for that specific signal. The handler runs in the context of the process - it uses the process's memory and stack. After the handler finishes, the process resumes whatever it was doing before the signal arrived.

If the process did not register a handler for the signal, the OS applies the default action for that signal. For most signals, the default action is one of: terminate the process (SIGTERM, SIGINT, SIGKILL), terminate and dump core (SIGSEGV, SIGFPE, SIGABRT), stop the process (SIGSTOP, SIGTSTP), continue a stopped process (SIGCONT), or ignore the signal (SIGCHLD, SIGURG).

sequenceDiagram
    participant Sender as Sender<br/>(another process<br/>or the OS)
    participant Kernel as OS Kernel
    participant Target as Target Process
    
    Sender->>Kernel: kill(pid, SIGTERM)
    Kernel->>Kernel: Set SIGTERM pending<br/>in target's PCB
    
    Note over Target: Process is running<br/>normally...
    
    Kernel->>Target: Deliver signal<br/>(at next scheduling point)
    
    alt Handler registered
        Target->>Target: Run SIGTERM handler<br/>(cleanup, save state)
        Target->>Target: Handler returns<br/>Resume normal execution<br/>(or call exit)
    else No handler (default action)
        Target->>Kernel: Process terminated<br/>(default for SIGTERM)
    end

This sequence diagram shows signal delivery. A sender calls kill with a PID and signal number. The kernel marks the signal as pending in the target process's PCB. When the target process is next scheduled, the kernel delivers the signal. If the process has a registered handler for that signal, the handler runs and the process resumes. If no handler is registered, the default action applies - for SIGTERM, the default is to terminate the process.

The Most Important Signals

You do not need to memorize all 30+ signals for an interview. But you should know the most commonly used ones and their purposes:

SIGTERM vs SIGKILL: The Polite Way and the Hard Way

This is the most important signal distinction for interviews and production systems.

SIGTERM (signal 15) is a polite request to terminate. The process can catch this signal and run a cleanup handler - close database connections, flush buffers, save state, notify other services, and then exit gracefully. This is the signal that kill sends by default (when you type kill 1234 without specifying a signal number).

SIGKILL (signal 9) is the nuclear option. The process cannot catch, block, or ignore it. The kernel terminates the process immediately, with no opportunity for cleanup. Open files may not be flushed. Database connections may not be closed. Temporary files may not be deleted. Locks may not be released. Use SIGKILL only when SIGTERM has failed - when the process is frozen and unresponsive to polite requests.

The correct shutdown sequence is: send SIGTERM, wait a reasonable time (5-30 seconds), and only send SIGKILL if the process has not exited. This is exactly what Kubernetes does when terminating a pod - it sends SIGTERM to the main process, waits for the terminationGracePeriodSeconds (default 30 seconds), and then sends SIGKILL if the process is still alive.

Docker follows the same pattern: docker stop sends SIGTERM, waits 10 seconds (configurable with -t), then sends SIGKILL. Understanding this pattern is important for building containerized services that shut down gracefully.

Interview: "What is the difference between SIGTERM and SIGKILL?"

"SIGTERM is a polite termination request - the process can catch it and perform cleanup before exiting. SIGKILL is an immediate, uncatchable kill - the kernel terminates the process with no opportunity for cleanup. The correct pattern is to send SIGTERM first, wait for the process to exit gracefully, and only send SIGKILL if it does not respond within a timeout.

This pattern is used by Kubernetes (sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL) and Docker (sends SIGTERM, waits 10 seconds, then SIGKILL). A well-written service catches SIGTERM, stops accepting new requests, finishes in-flight requests, flushes buffers, closes connections, and exits cleanly."

Writing a Signal Handler

A process registers a signal handler using the signal() or sigaction() system call. When the signal arrives, the OS interrupts the process and calls the handler function. After the handler returns, the process resumes where it left off.

import signal
import os
import sys

# Register a handler for SIGTERM
def handle_sigterm(signum, frame):
    print(f"\nReceived signal {signum} (SIGTERM)")
    print("Performing graceful shutdown:")
    print("  - Closing database connections...")
    print("  - Flushing write buffers...")
    print("  - Saving application state...")
    print("  - Cleanup complete. Exiting.")
    sys.exit(0)

# Register a handler for SIGINT (Ctrl+C)
def handle_sigint(signum, frame):
    print(f"\nReceived signal {signum} (SIGINT - Ctrl+C)")
    print("Interrupted! Cleaning up...")
    sys.exit(1)

signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigint)

print(f"Process {os.getpid()} is running with signal handlers installed.")
print(f"Default SIGTERM action: terminate")
print(f"Our SIGTERM action: graceful shutdown with cleanup")
print(f"To test: run 'kill {os.getpid()}' from another terminal")
print(f"(In this demo, we'll just show the handlers are installed)")

The Danger Zone: What You Must NOT Do in a Signal Handler

Signal handlers look simple, but they are one of the most treacherous areas of systems programming. The problem is that a signal can arrive at any moment - in the middle of a malloc() call, in the middle of a printf() call, in the middle of updating a data structure. If the signal handler calls malloc() or printf() itself, it might corrupt the internal state of those functions because they were already in the middle of executing.

Functions that are safe to call inside a signal handler are called async-signal-safe functions. The POSIX standard defines a specific list of async-signal-safe functions - mostly simple system calls like write(), _exit(), open(), close(), and a few others. Notably, printf(), malloc(), free(), and pthread_mutex_lock() are NOT async-signal-safe. Calling them from a signal handler can cause deadlocks, memory corruption, and crashes that are extremely difficult to debug because they depend on the exact timing of the signal delivery.

The safe pattern for signal handlers is to do as little as possible. Set a global volatile flag (of type sig_atomic_t) and return. The main loop of the program checks the flag periodically and performs the actual cleanup in the main execution context, where all functions are safe to call.

// Safe signal handler pattern (C)
volatile sig_atomic_t shutdown_requested = 0;

void handle_sigterm(int signum) {
    // Just set the flag - do NOT call printf, malloc, or anything complex
    shutdown_requested = 1;
}

int main() {
    signal(SIGTERM, handle_sigterm);
    
    while (!shutdown_requested) {
        // Main work loop
        process_next_request();
    }
    
    // Cleanup happens here, in the main context, where everything is safe
    close_database_connections();
    flush_buffers();
    save_state();
    return 0;
}

Warning: The cardinal rule of signal handlers: Do not do complex work in a signal handler. Set a flag and return. Perform the actual cleanup in the main program loop. Calling non-async-signal-safe functions (printf, malloc, mutex operations) from a signal handler can cause deadlocks and crashes that are nearly impossible to reproduce. This is one of the most common signal-related bugs in production C and C++ code.

Signals in Production Systems

Graceful shutdown in web servers. Nginx, Apache, and Gunicorn all use SIGTERM for graceful shutdown and SIGHUP for configuration reload. When Nginx receives SIGHUP, it re-reads its configuration file, starts new worker processes with the new configuration, and gracefully stops the old workers after they finish handling in-flight requests. This enables zero-downtime configuration changes - a critical operational capability.

Log rotation. Many services use SIGUSR1 to trigger log file rotation. The process closes its current log file, opens a new one, and the log rotation tool (like logrotate) can safely rename and compress the old file. This avoids the need to restart the service just to rotate logs.

Debugging with SIGQUIT. In Java, sending SIGQUIT (kill -3) to a JVM process prints a thread dump to stderr - a stack trace of every thread. This is an essential debugging tool for diagnosing deadlocks, hangs, and performance issues in Java applications. Go has a similar feature - sending SIGQUIT prints goroutine stack traces.

SIGCHLD and zombie prevention. As covered in Module 01, when a child process terminates, the OS sends SIGCHLD to the parent. If the parent installs a SIGCHLD handler that calls waitpid(-1, &status, WNOHANG), it can clean up zombie processes asynchronously without blocking. This is the standard pattern for server processes that fork worker children.

Container orchestration and signals. Kubernetes and Docker rely heavily on signals for container lifecycle management. When Kubernetes needs to stop a pod, it sends SIGTERM to PID 1 inside the container. The application should catch SIGTERM and begin graceful shutdown - stop accepting new connections, drain in-flight requests, close database connections, and exit. If the application does not exit within the grace period, Kubernetes sends SIGKILL.

A common production bug occurs when the container's entrypoint script runs the application as a child process without forwarding signals. The SIGTERM reaches the shell script (PID 1), but the shell ignores it by default. The application never receives the termination signal and continues running until SIGKILL arrives 30 seconds later - an ungraceful shutdown that can cause data loss, dropped connections, and inconsistent state. The fix is to use exec in the entrypoint script to replace the shell with the application process, making the application PID 1 and the direct recipient of signals.

This signal-forwarding problem is common enough that many Dockerfiles and entrypoint scripts include explicit signal handling. Understanding it demonstrates production container experience in interviews.

graph TD
    subgraph SIGNALS_BY_USE["Signals Grouped by Use Case"]
        subgraph TERM["Process Termination"]
            SIGTERM_N["SIGTERM - Polite shutdown"]
            SIGKILL_N["SIGKILL - Force kill"]
            SIGINT_N["SIGINT - Ctrl+C"]
        end
        
        subgraph LIFECYCLE["Process Lifecycle"]
            SIGCHLD_N["SIGCHLD - Child exited"]
            SIGSTOP_N["SIGSTOP - Pause process"]
            SIGCONT_N["SIGCONT - Resume process"]
        end
        
        subgraph ERROR["Error Signals"]
            SIGSEGV_N["SIGSEGV - Bad memory access"]
            SIGFPE_N["SIGFPE - Divide by zero"]
            SIGPIPE_N["SIGPIPE - Broken pipe"]
        end
        
        subgraph CUSTOM["Application-Defined"]
            SIGUSR1_N["SIGUSR1 - Custom (log rotate, etc)"]
            SIGHUP_N["SIGHUP - Reload config"]
        end
    end

This diagram groups common signals by use case. Process termination signals include SIGTERM for polite shutdown, SIGKILL for force kill, and SIGINT for Ctrl+C. Process lifecycle signals include SIGCHLD when a child exits, SIGSTOP to pause, and SIGCONT to resume. Error signals include SIGSEGV for bad memory access, SIGFPE for divide by zero, and SIGPIPE for broken pipes. Application-defined signals include SIGUSR1 for custom purposes like log rotation and SIGHUP for configuration reload.

Signals vs Other IPC Mechanisms

Signals are fundamentally different from pipes, message queues, shared memory, and sockets. Those mechanisms transfer data - bytes, messages, memory regions. Signals transfer notifications - a signal number, nothing more. You cannot send a string or a data structure through a signal. You can only say "event X happened."

This makes signals the wrong choice for data transfer but the right choice for several specific patterns: notifying a process that it should shut down (SIGTERM), notifying a process that its configuration has changed (SIGHUP), notifying a parent that a child has exited (SIGCHLD), and notifying a process that an error occurred (SIGSEGV, SIGPIPE). In each case, the signal carries a simple notification, and the process decides what to do about it.

For data-carrying communication between processes, use pipes, sockets, or shared memory. For event notifications, use signals. Knowing which IPC mechanism to use for which purpose demonstrates the kind of systems-level judgment that interviewers value.

Exercise: Starter: In a terminal, start a sleep 300 command in the background (sleep 300 &). Note its PID. Send it SIGTERM (kill [PID]). Check if it is still running (ps -p [PID]). Now start another sleep 300 & and send it SIGSTOP (kill -STOP [PID]). Check its state in ps - it should be stopped (T state). Send SIGCONT (kill -CONT [PID]) to resume it.

Intermediate: Write a Python program that registers handlers for SIGTERM, SIGINT, and SIGUSR1. Each handler should print which signal was received and set a flag. The main loop should check the flags and perform different actions: SIGTERM triggers graceful shutdown, SIGINT triggers immediate exit, and SIGUSR1 triggers a status report (print current state). Test by sending signals from another terminal.

Advanced: Research the self-pipe trick - a technique for safely integrating signal handling with an event loop (select/poll/epoll). Why is this necessary? What problem does it solve that a simple signal handler with a global flag does not? Implement a simple version: create a pipe, write a byte to the pipe in the signal handler, and read from the pipe in the event loop to detect the signal.

Back to Operating Systems