Free lesson · Node.js Deep Dive

The Streams Mental Model

Easy7 min readnodejsjavascriptbackend

Lesson 7.1 - The Streams Mental Model

Before any API, the mental model.

A stream is a sequence of bytes (or, in some cases, objects) that flows over time. You don't have all the data at once - you have it piece by piece, with gaps in time between pieces. A stream is the abstraction for "data that arrives over time" and "data that goes somewhere over time."

This is the core idea. Everything else in Module 7 - readable, writable, duplex, transform, pipe, pipeline, backpressure - is mechanics for working with this abstraction.

In contrast, the non-stream model is "buffer everything in memory, then process." Read the whole file. Read the whole HTTP body. Read the entire database query result. Hold all of it in RAM. Then operate.

The non-stream model is simpler. For small data, it's the right choice - it's easier to read and reason about. But it fails for large data in two ways: you can run out of memory, and you wait for the whole thing before you can start processing.

Streams solve both. They process data as it arrives, in constant memory, without waiting for completion.

Three Roles: Source, Sink, Transformation

Every stream has a role in a data flow:

flowchart LR
    Src[Source<br/>Readable]
    Tr[Transformation<br/>Transform / Duplex]
    Snk[Sink<br/>Writable]

    Src -->|data| Tr
    Tr -->|data| Snk

    style Src fill:#34d399,stroke:#10b981,color:#000
    style Tr fill:#fbbf24,stroke:#f59e0b,color:#000
    style Snk fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram showing the three roles of streams in a pipeline. On the left is a source - a Readable stream that produces data, like a file being read or a network socket receiving data. In the middle is a transformation - a Transform or Duplex stream that takes data in and emits modified data, like a gzip compressor or a JSON parser. On the right is a sink - a Writable stream that consumes data, like a file being written or a network response being sent. Data flows from source through transformations to sink.

Source (Readable stream). Produces data. Examples: fs.createReadStream reads from a file; http.IncomingMessage (the request object in a server) provides the request body; process.stdin reads from standard input.

Sink (Writable stream). Consumes data. Examples: fs.createWriteStream writes to a file; http.ServerResponse (the response object in a server) sends data to the client; process.stdout writes to standard output.

Transformation (Transform stream, a special kind of Duplex). Both consumes and produces data, applying some operation. Examples: zlib.createGzip consumes bytes and produces gzipped bytes; crypto.createCipheriv consumes plaintext and produces ciphertext.

A typical pipeline has one source, zero or more transformations, and one sink. They're connected so data flows from one to the next.

There's also Duplex streams - both readable and writable, but the two halves are independent (not transformations of each other). A TCP socket is a Duplex: you can write to it (sending data to the peer) and read from it (receiving data from the peer), but the data you read isn't a transformation of what you wrote.

The Mental Model: A Pipe Network

The clearest analogy: a stream is a section of pipe in a water-distribution network. Water enters from one end and exits the other. The pipe has a fixed capacity - water can only flow at the rate the smallest section allows. If the consumer's tap is closed and the source keeps pumping, water has to go somewhere; it pools up, and if the pool overflows, you have a problem.

This analogy captures every important property of streams:

Data flows in one direction (in a Readable; both directions in a Duplex).
The pipe has finite buffer space (the stream's "internal buffer").
If the source is faster than the sink, data pools up in the buffer.
If the buffer overflows... that's the problem. We'll cover this in Lesson 7.4 (backpressure).

When something goes wrong in a stream pipeline, it's almost always because of the buffer-overflow scenario: data accumulated faster than it was consumed, and either memory ballooned or data was lost.

Streams in Practice

To make this concrete, let me show a complete stream pipeline.

const { pipeline } = require('node:stream/promises');
const fs = require('node:fs');
const zlib = require('node:zlib');

await pipeline(
  fs.createReadStream('input.txt'),    // source: read from file
  zlib.createGzip(),                   // transform: gzip-compress
  fs.createWriteStream('input.txt.gz') // sink: write to file
);
console.log('Done');

This reads a file, compresses it, and writes the result to a new file. With streams, the file is processed in chunks. At any given moment, the entire pipeline holds maybe 16 KB of data total - regardless of whether the file is 100 KB, 100 MB, or 100 GB.

Without streams:

const input = await fs.promises.readFile('input.txt');     // entire file in memory
const compressed = zlib.gzipSync(input);                    // entire compressed file in memory
await fs.promises.writeFile('input.txt.gz', compressed);

For a 100 GB file, this OOMs. The stream version doesn't. That's the point.

The Buffer Inside a Stream

Each stream has an internal buffer - a queue of chunks waiting to be consumed (for readable) or to be flushed (for writable). The default size is 16 KB for byte streams.

The buffer is what enables decoupling between source and sink. The source can produce ahead of the sink, up to the buffer size. Beyond that, the source has to slow down - this is backpressure.

For a Readable stream:

The buffer fills as the source reads from the underlying resource.
Consumers drain the buffer.
When the buffer is full and there's more to read, the source pauses.

For a Writable stream:

Calls to .write(chunk) add chunks to the buffer.
The buffer drains as the underlying resource accepts the data.
When the buffer is full, .write() returns false, signaling backpressure.

The buffer size is called the high water mark - it's the threshold at which the stream applies backpressure. You can configure it (new Writable({ highWaterMark: 1024 * 1024 })) but the default is usually fine.

Chunks: What Flows Through a Stream

The data flowing through a byte stream is a sequence of Buffer objects. Each chunk is some number of bytes - could be 1 byte, could be 16 KB. The chunks have no inherent meaning to the stream; they're just bytes.

This means: chunk boundaries don't correspond to logical message boundaries. If you write "Hello, world!" to a stream as a single string, the consumer might receive it as two chunks: "Hello, " and "world!". Or three. Or one. The stream doesn't preserve write boundaries.

This trips up engineers who assume each 'data' event corresponds to one logical message. For a line-oriented protocol (HTTP, SMTP, Redis's RESP), you have to reassemble - accumulate chunks, look for delimiters, parse what's complete, save what's not.

There's also object mode streams, where the chunks are arbitrary JavaScript values (not just bytes). For example, a database query stream might emit one chunk per row (each chunk is a row object). Object mode preserves boundaries - each chunk is a discrete value. We'll cover object mode in Lesson 7.6.

Three Generations of the Stream API

A bit of history. The stream API has gone through three major versions:

Streams 1 ("classic streams", Node 0.x). The original streams API. Buggy, hard to use correctly. Don't write code against it; it's been retired.

Streams 2 (Node 0.10+, around 2013). A major redesign. Introduced the Readable, Writable, Duplex, Transform base classes with the modern internal API (_read, _write, _transform). This is the foundation of modern streams.

Streams 3 (Node 0.11+). A small refinement. Added the pipe() method, the flowing/paused modes, and other ergonomics. This is what we use today.

Async iterators on streams (Node 10+). Not really a new API; an additional way to consume Readable streams. for await (const chunk of readable) works on any Readable. Often the cleanest way to consume.

For new code, target Streams 3 with async iterators. The classic event-based API (stream.on('data'), stream.on('end')) still works and is sometimes the right tool, but async iterators are usually clearer.

What I Want You to Take From This Lesson

Four things.

One: Streams are sequences of data over time. Use them when you have lots of data, when data arrives gradually, or when you want constant memory regardless of total size.

Two: Three roles - source, sink, transformation. Every stream is a Readable, a Writable, or both (Duplex/Transform). Pipelines connect them.

Three: Internal buffers decouple source from sink. The high water mark is the threshold for backpressure. We'll cover backpressure in 7.4.

Four: Byte streams don't preserve write boundaries. Chunks are arbitrary slices of the byte sequence. For line-oriented or message-oriented protocols, you have to reassemble. Object mode preserves boundaries.

Try It

Exercise: Warmup (10 minutes). Pick a file on your system that's at least 10 MB. Read it three ways: (a) fs.readFile, (b) fs.createReadStream then accumulate all chunks with for await...of, (c) pipeline(fs.createReadStream(...), fs.createWriteStream(...)). Compare your code, your memory usage (via process.memoryUsage()), and execution time.

Exercise: Core (15 minutes). Take a file. Use pipeline to read it, gzip it, and write the compressed version. Verify the result with gunzip from the command line.

Exercise: Core (15 minutes). Write a server that streams /dev/urandom to the client (on macOS/Linux). The server should be able to send gigabytes without blowing up its memory. Use pipeline(fs.createReadStream('/dev/urandom'), res). Verify by downloading with curl: curl http://localhost:3000/random -o random.bin and let it run for a few seconds. The server's memory should stay flat.

Exercise: Stretch (30 minutes). Read the Node.js streams documentation introduction. Don't worry about details yet - just read the top sections to internalize the terminology. We'll revisit specifics in 7.2-7.8.

What's Next

Lesson 7.2 covers Readable streams in detail - both the modern async-iterator API and the classic event-based API, with worked examples of each.

Back to Node.js Deep Dive