Free lesson · Operating Systems

Terminal 1 (writer):

Easy11 min readoperating-systemsosconcurrency

The Simplest Way Two Processes Can Talk

Pipes are the oldest and simplest IPC mechanism in Unix. When you type ls | grep txt in a terminal, you are using a pipe. The | symbol tells the shell to create a pipe - a unidirectional data channel - between the ls process and the grep process. Whatever ls writes to its standard output flows through the pipe and appears as the standard input of grep.

Pipes are everywhere in Unix systems. Shell scripting depends on them. Build systems chain commands through them. Log processing pipelines use them to filter and transform data in stages. Understanding how pipes work at the OS level is essential for answering the interview classic: "What happens when you type ls | grep txt in a terminal?"

How Pipes Work Internally

A pipe is a kernel-managed buffer - a region of memory inside the kernel that acts as a queue between two processes. One process writes data into the pipe. The other process reads data from the pipe. The kernel manages the buffer, handles synchronization, and ensures that data flows in order.

When a process calls the pipe() system call, the kernel creates the buffer and returns two file descriptors: one for the read end (fd[0]) and one for the write end (fd[1]). These file descriptors work just like file descriptors for regular files - you use read() and write() to transfer data through them. This is the Unix philosophy of "everything is a file" in action - a pipe looks and behaves like a file from the programmer's perspective.

The pipe buffer has a fixed size - typically 64 KB on Linux (16 pages of 4 KB each). This size limit creates natural flow control between the producer and consumer:

If the pipe buffer is full (the writer is faster than the reader), the write() call blocks until the reader reads some data and makes space. This prevents the writer from overwhelming the reader.

If the pipe buffer is empty (the reader is faster than the writer), the read() call blocks until the writer writes some data. This makes the reader wait rather than consuming nothing.

If the write end is closed (the writer has finished), the reader gets an end-of-file indication (read() returns 0). This is how the reader knows the writer is done.

If the read end is closed (the reader has exited or closed its end), the writer receives a SIGPIPE signal, which by default terminates the writing process. This prevents the writer from wasting CPU time writing data that nobody will read.

graph LR
    subgraph PIPE_INTERNAL["Pipe Internals"]
        WRITER["Writer Process<br/>────<br/>fd[1] (write end)"]
        BUFFER["Kernel Buffer<br/>────<br/>64 KB on Linux<br/>FIFO order<br/>Blocks when full/empty"]
        READER["Reader Process<br/>────<br/>fd[0] (read end)"]
    end
    
    WRITER -->|"write(fd[1], data)"| BUFFER
    BUFFER -->|"read(fd[0], buf)"| READER

This diagram shows the internal structure of a Unix pipe. The writer process writes data through file descriptor fd[1] into a kernel-managed buffer of 64 KB. The reader process reads data from the buffer through file descriptor fd[0]. Data flows in FIFO order. The write call blocks if the buffer is full, and the read call blocks if the buffer is empty.

How the Shell Uses Pipes

When you type ls | grep txt, the shell performs a precise sequence of operations that combines fork, exec, and pipe creation. Understanding this sequence is one of the most commonly tested OS concepts in interviews.

Step 1: The shell calls pipe(). The kernel creates the pipe buffer and returns two file descriptors: fd[0] (read end) and fd[1] (write end). At this point, the shell process holds both ends of the pipe.

Step 2: The shell calls fork() to create the first child (for ls). The child inherits copies of the shell's file descriptors, including both pipe ends. The child redirects its standard output (fd 1) to the pipe's write end (fd[1]) using dup2(), closes the original pipe file descriptors (it no longer needs them directly), and calls exec("ls"). Now ls is running with its stdout connected to the pipe.

Step 3: The shell calls fork() again to create the second child (for grep). This child redirects its standard input (fd 0) to the pipe's read end (fd[0]) using dup2(), closes the original pipe file descriptors, and calls exec("grep", "txt"). Now grep is running with its stdin connected to the pipe.

Step 4: The shell closes both pipe ends. The shell itself does not use the pipe - it only created it for its children. Closing the pipe ends is important: if the shell keeps the write end open, grep will never see end-of-file because the write end is still open somewhere.

Step 5: The shell waits for both children to finish. It calls waitpid() for each child.

sequenceDiagram
    participant Shell as Shell Process
    participant Kernel as OS Kernel
    participant LS as ls Process
    participant GREP as grep Process
    
    Shell->>Kernel: pipe() → fd[0], fd[1]
    
    Shell->>Kernel: fork() → child 1
    
    Note over LS: Child 1
    LS->>LS: dup2(fd[1], STDOUT)<br/>close(fd[0]), close(fd[1])
    LS->>Kernel: exec("ls")
    Note over LS: ls writes to pipe
    
    Shell->>Kernel: fork() → child 2
    
    Note over GREP: Child 2
    GREP->>GREP: dup2(fd[0], STDIN)<br/>close(fd[0]), close(fd[1])
    GREP->>Kernel: exec("grep", "txt")
    Note over GREP: grep reads from pipe
    
    Shell->>Shell: close(fd[0]), close(fd[1])
    Shell->>Kernel: waitpid(child1)<br/>waitpid(child2)
    
    Note over LS: ls writes filenames<br/>to stdout (→ pipe)
    Note over GREP: grep reads from stdin (← pipe)<br/>filters for "txt"

This sequence diagram shows how the shell implements ls pipe grep. The shell creates a pipe, getting two file descriptors. It forks child 1, which redirects its stdout to the pipe's write end and execs ls. It forks child 2, which redirects its stdin to the pipe's read end and execs grep. The shell closes its own pipe ends and waits for both children. ls writes filenames to the pipe, and grep reads them from the pipe, filtering for lines containing txt.

Interview: "What happens when you type ls | grep txt in a terminal?"

"The shell creates a pipe using the pipe() system call, which returns two file descriptors - one for reading and one for writing. The shell forks two child processes. The first child redirects its stdout to the pipe's write end using dup2(), closes the original pipe descriptors, and exec's the ls program. The second child redirects its stdin to the pipe's read end, closes the pipe descriptors, and exec's grep with the argument 'txt'.

The shell closes its own copies of the pipe ends and waits for both children to finish. ls writes its output to stdout, which goes through the pipe to grep's stdin. grep reads each line, checks if it contains 'txt', and writes matching lines to its own stdout (the terminal).

The pipe is a kernel-managed buffer - typically 64 KB on Linux. If ls produces data faster than grep can consume it, write() blocks until grep reads. If grep reads faster than ls produces, read() blocks until ls writes."

Unnamed Pipes vs Named Pipes (FIFOs)

The pipe we have been discussing is an unnamed pipe (also called an anonymous pipe). It exists only as long as the processes using it are alive, and it can only be used between related processes - typically a parent and its children, because the file descriptors are inherited through fork(). Unrelated processes cannot use an unnamed pipe because they have no way to get the file descriptors.

A named pipe (also called a FIFO - First In First Out) solves this limitation. A named pipe is a special file on the file system - it has a path, permissions, and an owner, just like a regular file. Any process that knows the path can open it and communicate through it.

You create a named pipe with the mkfifo command:

mkfifo /tmp/my_pipe

This creates a special file at /tmp/my_pipe. Now any two processes can communicate through it:

# Terminal 1 (writer):
echo "Hello from writer" > /tmp/my_pipe

# Terminal 2 (reader):
cat < /tmp/my_pipe
# Output: Hello from writer

Named pipes behave exactly like unnamed pipes in terms of buffering and blocking - the kernel manages a buffer, writes block when the buffer is full, and reads block when the buffer is empty. The only difference is that named pipes have a file system presence, so unrelated processes can use them by opening the same path.

import os
import stat

# Check if a named pipe exists (demonstration)
pipe_path = "/tmp/demo_pipe"

# In a real system, you would use mkfifo:
# os.mkfifo(pipe_path)

# Let's demonstrate unnamed pipe instead
read_end, write_end = os.pipe()
print(f"Pipe created:")
print(f"  Read end:  fd {read_end}")
print(f"  Write end: fd {write_end}")

# Write to the pipe
message = b"Hello through the pipe!"
os.write(write_end, message)
print(f"  Wrote {len(message)} bytes to pipe")

# Read from the pipe
data = os.read(read_end, 1024)
print(f"  Read from pipe: {data.decode()}")

os.close(read_end)
os.close(write_end)
print("  Pipe closed")

When to Use Named Pipes vs Unnamed Pipes

Both types are unidirectional - data flows in one direction. If you need bidirectional communication, you need two pipes (one for each direction) or a different IPC mechanism like sockets.

Limitations of Pipes

Pipes are simple and elegant, but they have limitations that push you toward more sophisticated IPC mechanisms:

Unidirectional. Data flows one way. For request-response communication (like a client sending a query and receiving a result), you need two pipes or a socket.

No message boundaries. A pipe is a byte stream - it does not preserve message boundaries. If the writer writes "Hello" and then "World", the reader might receive "HelloWorld" in a single read, or "Hell" and "oWorld" in two reads. If you need message boundaries, you have to implement your own framing protocol (length-prefixed messages, delimiters, etc.) or use message queues.

Same-machine only. Pipes work between processes on the same machine. For communication across machines, you need sockets.

Limited buffer. The 64 KB buffer can become a bottleneck for high-throughput data transfer. If the producer and consumer operate at very different speeds, the small buffer causes frequent blocking.

Despite these limitations, pipes remain the right tool for their use case: simple, sequential data flow between processes on the same machine. Shell pipelines, build systems, and log processing are all perfectly served by pipes.

Multi-Stage Pipelines

Pipes become especially powerful when you chain multiple stages together. The command cat access.log | grep "500" | awk '{print

This pipeline architecture has a key advantage: all stages run concurrently. While grep is filtering lines, awk is extracting fields from lines that grep already passed through, and sort is accumulating lines from awk. The pipe buffers between each pair of processes allow each stage to work at its own pace, with the kernel managing flow control automatically.

The concurrency of pipeline stages is a form of parallelism that uses multiple CPU cores. On a 4-core machine, four pipeline stages can genuinely run simultaneously on different cores. This is why complex shell pipelines can process large files faster than you might expect - the work is distributed across stages that run in parallel, connected by kernel-managed buffers.

However, pipelines have a throughput bottleneck: the slowest stage determines the overall throughput. If sort takes 10 seconds to process its input while every other stage finishes in 1 second, the pipeline takes at least 10 seconds. This is analogous to an assembly line in manufacturing - the slowest station determines the overall production rate.

Performance Characteristics of Pipes

Pipe performance is surprisingly good for a kernel-mediated mechanism. Here are the key performance characteristics:

Throughput. On modern Linux, a pipe can transfer several GB/s between two processes on the same machine. The 64 KB buffer means data is being copied in and out of kernel space frequently, but modern CPUs can do memory copies at tens of GB/s, so the copy overhead is manageable.

Latency. The latency of a pipe write followed by a pipe read is typically 5-20 microseconds - the cost of two system calls (write + read) plus a memory copy through the kernel buffer. This is much faster than a TCP socket round-trip (which adds protocol processing overhead) but slower than shared memory (which avoids kernel involvement entirely).

Context switches. When a pipe write fills the buffer, the writing process blocks and the kernel wakes up the reading process. When a pipe read empties the buffer, the reading process blocks and the kernel wakes up the writing process. This ping-pong of context switches adds overhead when the buffer size is small relative to the data transfer rate. The splice() system call on Linux can reduce this overhead by moving data between pipes and file descriptors without copying through user space.

Tip: For system design interviews: When discussing data pipelines or stream processing, you can reference Unix pipe architecture as the conceptual foundation. "The processing pipeline works like a chain of Unix pipes - each stage transforms data and passes it to the next, with buffered channels between stages for flow control and concurrent execution." This analogy resonates with interviewers because it connects a familiar concept (shell pipes) to a distributed system design.

Pipes in Modern Applications

Although pipes are an old mechanism (they date back to Unix v3 in 1973), they remain widely used in modern systems:

Build systems. Tools like Make, Bazel, and Gradle use pipes to capture the stdout and stderr of build commands, allowing the build system to display output, detect errors, and log results.

Container runtimes. Docker and containerd use pipes to connect to the stdin, stdout, and stderr of containerized processes. When you run docker logs -f container_id, Docker reads from a pipe connected to the container's stdout.

Process managers. Systemd and supervisord use pipes to capture the output of managed services. The pipe connects the service's stdout to the process manager's logging system, which writes the output to the journal or log files.

Testing frameworks. Testing tools capture program output through pipes to compare it against expected output. The pipe lets the test framework read exactly what the program wrote, including timing and ordering of output.

Exercise: Starter: In a terminal, run ls -la /proc/self/fd. This shows the file descriptors open in the current shell process. Can you identify stdin (fd 0), stdout (fd 1), and stderr (fd 2)? Now run ls -la /proc/self/fd | cat and check fd 1 - it should now point to a pipe instead of the terminal.

Intermediate: Write a Python program that creates a pipe using os.pipe(), forks a child process, has the child write a message through the pipe, and has the parent read the message. Verify that the message arrives correctly. What happens if the parent tries to read before the child writes?

Advanced: Create a named pipe using mkfifo. Write a Python script that reads from the named pipe and another script that writes to it. Run them in separate terminals. Observe the blocking behavior - the reader blocks until the writer opens the pipe, and vice versa. Now try writing faster than reading - what happens when the buffer fills up?

Back to Operating Systems