Free lesson · Operating Systems

Create a shared memory block

Easy11 min readoperating-systemsosconcurrency

The Fastest IPC Mechanism

Every IPC mechanism you have learned so far - pipes, message queues - involves the kernel mediating every data transfer. When Process A sends data through a pipe, the kernel copies the data from Process A's address space into the kernel's pipe buffer. When Process B reads from the pipe, the kernel copies the data from the pipe buffer into Process B's address space. That is two copies and two system calls for every chunk of data.

Shared memory eliminates this overhead entirely. Two processes map the same region of physical memory into their respective virtual address spaces. Once the mapping is established (via a one-time system call), both processes can read and write the shared region directly - no system calls, no kernel involvement, no data copying. Communication is just memory reads and writes, which take nanoseconds.

This makes shared memory the fastest IPC mechanism available. It is used in high-performance systems where the overhead of kernel-mediated IPC is unacceptable - database buffer pools, multimedia processing pipelines, scientific computing, financial trading systems, and real-time data sharing between cooperating processes.

But shared memory is also the most dangerous IPC mechanism. Because both processes can modify the shared region at any time without kernel mediation, all the concurrency problems from Module 05 apply: race conditions, data corruption, and the need for explicit synchronization. The kernel does not protect you - it just gives you a shared memory region and walks away.

How Shared Memory Works

Setting up shared memory involves three steps: creating the shared memory segment, attaching it to each process's address space, and (eventually) detaching and destroying it.

Step 1: Create the shared memory segment. One process calls shmget() (System V) or shm_open() (POSIX) to create a named shared memory region of a specified size. The kernel allocates physical memory pages for the segment but does not map them into any process's address space yet.

Step 2: Attach the segment to each process. Each process that wants to use the shared memory calls shmat() (System V) or mmap() (POSIX) to map the shared memory segment into its own virtual address space. After this call, the process has a pointer to the shared memory - it can read and write through this pointer as if it were a regular malloc'd buffer.

Step 3: Use the shared memory. Both processes read and write through their pointers. The pointers in each process point to different virtual addresses (each process has its own address space), but those virtual addresses are mapped to the same physical memory pages. A write by Process A is immediately visible to Process B because they are literally writing to the same physical memory.

Step 4: Detach and destroy. When a process is done, it calls shmdt() (System V) or munmap() (POSIX) to unmap the shared memory from its address space. When all processes have detached and the segment is no longer needed, it is destroyed with shmctl(IPC_RMID) (System V) or shm_unlink() (POSIX).

graph TD
    subgraph PA_SPACE["Process A's Virtual Address Space"]
        PA_CODE["Code"]
        PA_HEAP["Heap"]
        PA_SHM["Shared Memory<br/>mapped at 0x7f001000"]
        PA_STACK["Stack"]
    end
    
    subgraph PB_SPACE["Process B's Virtual Address Space"]
        PB_CODE["Code"]
        PB_HEAP["Heap"]
        PB_SHM["Shared Memory<br/>mapped at 0x7f002000"]
        PB_STACK["Stack"]
    end
    
    subgraph PHYSICAL["Physical Memory"]
        PHYS_SHM["Shared Pages<br/>Physical Address 0x1A000"]
    end
    
    PA_SHM -->|"same physical pages"| PHYS_SHM
    PB_SHM -->|"same physical pages"| PHYS_SHM

This diagram shows how shared memory works at the virtual memory level. Process A's address space has the shared memory mapped at virtual address 0x7f001000. Process B's address space has the same shared memory mapped at virtual address 0x7f002000. Both virtual addresses point to the same physical memory pages at physical address 0x1A000. When either process writes to its mapped address, the other process sees the change immediately because they share the same physical pages.

The Synchronization Problem

Shared memory gives you raw speed, but it transfers the responsibility for correctness to the programmer. Without synchronization, two processes writing to shared memory simultaneously will produce corrupted data - exactly the same race conditions that occur with multi-threaded access to shared variables.

Consider a shared counter. Process A reads the value (42), increments it in a local register (43), and prepares to write it back. Before A writes, Process B reads the same value (42), increments it (43), and writes 43. Now A writes 43. The counter should be 44 (two increments), but it is 43 - one increment was lost.

This is the same race condition from Module 02 (threads), but now it is between processes rather than threads. The fix is also the same: synchronization using mutexes, semaphores, or atomic operations. For shared memory between processes, the most common synchronization mechanisms are POSIX semaphores (which can be configured for inter-process use) and POSIX mutexes (which can be placed in the shared memory region itself and configured with the PTHREAD_PROCESS_SHARED attribute).

sequenceDiagram
    participant PA as Process A
    participant SHM as Shared Memory<br/>(counter = 42)
    participant PB as Process B
    
    PA->>SHM: Read counter (42)
    Note over PA: Register = 42
    
    PB->>SHM: Read counter (42)
    Note over PB: Register = 42
    
    Note over PA: Increment: 42 + 1 = 43
    Note over PB: Increment: 42 + 1 = 43
    
    PA->>SHM: Write 43
    PB->>SHM: Write 43
    
    Note over SHM: Counter = 43<br/>Expected: 44<br/>RACE CONDITION!

This sequence diagram shows a race condition in shared memory. Both Process A and Process B read the shared counter value of 42. Both increment their local copy to 43. Both write 43 back to shared memory. The final value is 43 instead of the expected 44 - one increment was lost because the operations were not synchronized.

import multiprocessing
import multiprocessing.shared_memory
import time

# Create a shared memory block
shm = multiprocessing.shared_memory.SharedMemory(
    create=True, size=100, name="demo_shm"
)

# Write data to shared memory
message = b"Hello from shared memory!"
shm.buf[:len(message)] = message

print(f"Shared memory name: {shm.name}")
print(f"Shared memory size: {shm.size} bytes")
print(f"Data written: {bytes(shm.buf[:len(message)]).decode()}")

# In a real application, another process would do:
# shm2 = shared_memory.SharedMemory(name="demo_shm")
# data = bytes(shm2.buf[:25])  # Read the same data!

# Cleanup
shm.close()
shm.unlink()
print("Shared memory cleaned up")

Warning: The cardinal rule of shared memory: Never use shared memory without synchronization unless you can guarantee that only one process writes and the other only reads, AND you do not care about read consistency (tearing is acceptable). In practice, always use a synchronization mechanism - a POSIX semaphore, a mutex in the shared region, or a lock file. The performance gain of shared memory is meaningless if the data is corrupted.

Shared Memory in Real Systems

Database buffer pools. PostgreSQL's shared buffer pool is the textbook example of shared memory in production. All PostgreSQL backend processes (one per client connection) share a single buffer pool - an in-memory cache of disk pages. When one backend reads a page from disk, it goes into the shared buffer pool, and all other backends can access it without re-reading from disk. The buffer pool is protected by lightweight locks (LWLocks) that allow concurrent reads but exclusive writes to individual buffer pages.

The performance impact is enormous. Without shared memory, each PostgreSQL backend would need its own cache of disk pages, consuming far more memory and losing the benefit of shared caching. With shared memory, a page that was read by one client is immediately available to all other clients - a crucial optimization for database workloads where many clients access the same tables.

Memory-mapped files (mmap()). The mmap() system call can map a file into a process's virtual address space. If two processes mmap the same file with the MAP_SHARED flag, they share the same physical pages - changes made by one process are visible to the other. This is a form of shared memory that is backed by a file on disk, providing both the speed of shared memory and the persistence of file storage.

Redis uses mmap() for its RDB persistence format - memory-mapping the data file allows Redis to access data with zero-copy efficiency. Databases like SQLite and MongoDB also use mmap() for accessing data files.

High-frequency trading. Financial trading systems use shared memory for ultra-low-latency communication between the market data feed handler (which receives price updates from exchanges) and the trading strategy process (which decides whether to buy or sell). The feed handler writes price updates to a shared memory ring buffer, and the strategy process reads them. The latency is nanoseconds - far too fast for any kernel-mediated IPC mechanism.

Chrome's rendering architecture. Chrome uses shared memory to pass rendered bitmap data from renderer processes to the browser process for display. The renderer draws the web page into a shared memory buffer, and the browser process reads from the same buffer to composite the final display. This avoids copying potentially large bitmap data through kernel buffers.

Memory-Mapped Files: Shared Memory Backed by Disk

The mmap() system call deserves special attention because it bridges shared memory and file I/O in a way that powers many production systems. When you call mmap() on a file with the MAP_SHARED flag, the OS maps the file's contents into your process's virtual address space. You can then read and write the file by reading and writing memory - no read() or write() system calls needed.

If two processes both mmap() the same file with MAP_SHARED, they share the same physical pages. Changes made by one process appear in the other process's mapping, and changes are eventually written back to the file by the OS (or immediately with msync()). This gives you three things simultaneously: shared memory for fast inter-process communication, persistence because the data is backed by a file, and lazy loading because the OS loads pages from disk only when they are accessed (via page faults).

This is how many databases access their data files. Instead of calling read() and write() for every database page, they mmap() the entire data file and access it through memory. The OS page cache handles caching - frequently accessed pages stay in RAM, and the database does not need to implement its own buffer management. SQLite, MongoDB (with the WiredTiger storage engine in memory-mapped mode), and LMDB all use this approach.

The tradeoff is control. With mmap(), the OS decides which pages to evict from RAM when memory is tight. The database cannot control eviction priorities - it cannot say "keep this index page in RAM because it is accessed frequently." Databases like PostgreSQL prefer managing their own buffer pool (using shmget()-style shared memory) precisely because they need fine-grained control over which pages stay in memory.

When mmap() Is the Right Choice

Use mmap() when you need to read a large file efficiently (the OS handles caching and prefetching for you), when you want shared memory that persists across process restarts, when multiple processes need to access the same file concurrently, or when you want to leverage the OS page cache rather than implementing your own. Avoid mmap() when you need precise control over cache eviction, when the file is on a network file system (NFS mmap has consistency challenges), or when the file changes size frequently (resizing mmap'd regions is complex).

Security Considerations for Shared Memory

Shared memory introduces security concerns that kernel-mediated IPC mechanisms handle automatically:

Access control. Shared memory segments have Unix-style permissions (owner, group, others) that control which processes can attach. But once a process attaches, it has direct access to the raw bytes - the kernel does not validate what the process reads or writes. A malicious process that attaches to a shared memory segment can read sensitive data (credentials, encryption keys) or corrupt data that other processes depend on.

Information leakage. When the kernel allocates physical pages for a shared memory segment, those pages might contain data from previously freed memory (from other processes). The kernel should zero-fill new pages to prevent information leakage, and modern kernels do this by default. But this is a concern in high-security environments where even residual data in memory is a risk.

Denial of service. A process with access to a shared memory segment can corrupt the data, causing other processes to crash or behave incorrectly. This is why shared memory is typically used between cooperating processes that trust each other - not between processes with different security domains. For communication between untrusted processes, kernel-mediated IPC (pipes, sockets) is preferred because the kernel validates and copies data at each boundary.

These security considerations explain why microservices and multi-tenant systems almost never use shared memory for inter-service communication. The isolation boundary between services is a security boundary, and shared memory breaks that boundary. Sockets and message brokers preserve it.

Performance Numbers: How Fast Is Shared Memory?

To give you concrete numbers you can reference in interviews:

Shared memory latency: A write followed by a read from shared memory (with proper memory barriers) takes approximately 10-100 nanoseconds - the cost of a cache line transfer between CPU cores. This is 100-1000x faster than a pipe round-trip (5-20 microseconds) and 10,000-100,000x faster than a TCP socket round-trip (50-500 microseconds for localhost).

Shared memory throughput: Because there is no data copying, shared memory throughput is limited only by memory bandwidth - typically 20-50 GB/s on modern hardware. Pipe throughput is 2-10 GB/s (limited by kernel buffer copies). Socket throughput is 1-5 GB/s (limited by protocol processing and copies).

Setup cost: Creating and attaching a shared memory segment takes 5-50 microseconds (system calls for shmget/shmat or shm_open/mmap). This is a one-time cost - amortized over millions of reads and writes, it is negligible.

These numbers make it clear why shared memory is the choice for high-performance, same-machine communication - the latency and throughput advantages are orders of magnitude better than any alternative.

System V vs POSIX Shared Memory

Interview: "What is shared memory and when would you use it?"

"Shared memory is an IPC mechanism where two processes map the same physical memory pages into their virtual address spaces. After a one-time setup system call, communication is just memory reads and writes - no kernel involvement, no data copying, nanosecond latency.

It is the fastest IPC mechanism available, but it requires explicit synchronization (mutexes or semaphores) to prevent race conditions. I would use it when processes need to exchange large amounts of data at high speed on the same machine - like a database buffer pool shared across backend processes, or a rendering pipeline where one process produces bitmap data that another process displays.

For simpler communication needs, I would use pipes or sockets instead - they are slower but safer because the kernel manages data integrity and ordering."

Shared Memory vs Other IPC Mechanisms

Exercise: Starter: Using Python's multiprocessing.shared_memory module, create a shared memory block, write a string to it, and read it back. Verify that the data persists in the shared memory even after you close and reopen it (using the name).

Intermediate: Create two Python scripts - a writer and a reader. The writer creates a shared memory block and writes a counter value to it, incrementing every second. The reader opens the same shared memory block and reads the counter every second. Run both simultaneously and verify that the reader sees the updates from the writer. What happens if both try to increment the counter simultaneously?

Advanced: Design a shared memory ring buffer for a producer-consumer scenario. The producer writes fixed-size messages to the next available slot in the ring buffer. The consumer reads from the oldest unread slot. Use atomic operations or a semaphore to synchronize access. Implement it in Python or C and measure the throughput - how many messages per second can you achieve? Compare this to pipe-based and socket-based implementations.

Back to Operating Systems