Writable Streams: Managing Data Flow with Backpressure

Have you ever tried to fill a bucket with water from a fire hose? The water comes out so fast that the bucket overflows, and water splashes everywhere. Now imagine if the bucket could somehow signal the hose: "Hey! Slow down! I'm getting full!"

That's exactly what happens in Node.js when you try to write data faster than your system can handle it. And just like our smart bucket, Node.js streams have a built-in mechanism to prevent this overflow. Today, we're going on a journey to discover how writable streams protect your applications from memory crashes using something called backpressure.

Quick Reference

When to use writable streams:

• Writing data to files on disk
• Sending data over network connections
• Processing data that arrives faster than you can handle it

The backpressure signal:

const canContinueWriting = writableStream.write(data);
if (!canContinueWriting) {
  // Buffer is full! Need to wait before writing more
}

Common use cases:

• File copying and processing
• HTTP response streaming
• Database bulk operations
• Real-time data processing

Key insight: ⚡ Writable streams use backpressure to prevent memory overflow by signaling when to pause data flow

---

What You Need to Know First

To get the most out of this guide, you should understand:

• Node.js fundamentals: How to create and run Node.js scripts, use require or import statements
• Basic readable streams: What streams are and how they deliver data in chunks (see Readable Streams guide)
• File system basics: How to work with fs module for reading and writing files
• JavaScript events: How event emitters work (on, emit, event handlers)

If you're not familiar with readable streams, we strongly recommend reading that guide first, as writable streams work closely with them.

What We'll Cover in This Article

By the end of this guide, you'll understand:

• What writable streams are and why they're essential
• How internal buffers manage data flow
• What backpressure is and why it prevents crashes
• The highWaterMark concept and buffer limits
• The relationship between readable and writable streams
• The dramatic difference between handling and ignoring backpressure

What We'll Explain Along the Way

Don't worry if you're unfamiliar with these—we'll explain them as we go:

• Internal buffers (visual explanation included)
• Memory overflow scenarios
• The boolean return value of write()
• Buffer state monitoring

---

The Problem: When Data Flows Too Fast

Let's start with a real-world scenario that many developers encounter.

Imagine you're building a file processing application. A user uploads a 500MB video file, and you need to copy it to a different location on your server. You write what seems like straightforward code:

import fs from "fs";

const readStream = fs.createReadStream("large-video.mp4");
const writeStream = fs.createWriteStream("backup/large-video.mp4");

readStream.on("data", (chunk) => {
  writeStream.write(chunk); // Just write each chunk as it arrives
});

readStream.on("end", () => {
  writeStream.end();
  console.log("File copied!");
});

This code looks reasonable, right? Read some data, write some data. What could go wrong?

Let me tell you what I discovered when I first wrote code like this: my application crashed. Not immediately, but after processing several large files. The memory usage kept climbing until Node.js ran out of memory and shut down.

Why did this happen? Let's discover the answer together.

---

Understanding the Speed Mismatch

Here's the key insight that changed how I think about data processing:

Reading is usually faster than writing.

Think about it in physical terms:

• Reading a book (scanning text with your eyes) is fast
• Copying that book by hand (writing every word) is much slower

The same principle applies to computer operations:

// Reading from disk is relatively fast
const readStream = fs.createReadStream("input.txt");
// Data comes in quickly: chunk, chunk, chunk, chunk...

// Writing to disk is slower (especially with formatting, encoding, etc.)
const writeStream = fs.createWriteStream("output.txt");
// Processing takes time: chunk... still writing... done... next chunk...

What happens when reading is faster than writing?

The data has to wait somewhere. That "somewhere" is called a buffer - a temporary storage area in memory. Let's visualize this:

Reading Speed: ████████████ (Fast)
Writing Speed: ████ (Slower)

Where does the extra data go? → Into a buffer (memory)

Now here's the critical question: What happens if we keep reading fast but writing slowly?

---

The Buffer: Your Stream's Temporary Storage

Let's discover how writable streams manage this speed difference using internal buffers.

What Is a Buffer in a Writable Stream?

When you create a writable stream, Node.js automatically creates an internal buffer for it. Think of this buffer as a waiting room:

// When you create a writable stream
const writeStream = fs.createWriteStream("output.txt", {
  highWaterMark: 16384, // Buffer size: 16 KB (default)
});

// Imagine the buffer as a waiting room with 16,384 seats
// [Empty seats: 16,384] [Occupied seats: 0]

Every time you write data, it goes into this waiting room:

writeStream.write("Hello"); // 5 bytes enter the waiting room

// Buffer state:
// [Empty seats: 16,379] [Occupied seats: 5]

The stream processes data from the buffer and writes it to the actual destination (file, network, etc.). As data is written, seats become available again.

Visualizing Buffer Flow

Let's see what happens step by step:

import fs from "fs";

// Create a writable stream with a SMALL buffer for demonstration
const writeStream = fs.createWriteStream("output.txt", {
  highWaterMark: 5, // Only 5 bytes of buffer space
});

// Let's watch the buffer fill up
console.log("Buffer space available:", writeStream.writableLength); // 0 bytes used

writeStream.write("a"); // Write 1 byte
console.log('After writing "a":', writeStream.writableLength); // 1 byte used

writeStream.write("b"); // Write 1 byte
console.log('After writing "b":', writeStream.writableLength); // 2 bytes used

writeStream.write("c"); // Write 1 byte
console.log('After writing "c":', writeStream.writableLength); // 3 bytes used

writeStream.write("d"); // Write 1 byte
console.log('After writing "d":', writeStream.writableLength); // 4 bytes used

const canWriteMore = writeStream.write("e"); // Write 1 byte
console.log('After writing "e":', writeStream.writableLength); // 5 bytes used
console.log("Can write more?", canWriteMore); // false - buffer is full!

Output:

Buffer space available: 0
After writing "a": 1
After writing "b": 2
After writing "c": 3
After writing "d": 4
After writing "e": 5
Can write more? false

Notice what happened when we wrote the 5th byte? The write() method returned false. This is the stream's way of saying: "My buffer is full! Please wait!"

The highWaterMark: Your Buffer's Capacity

The highWaterMark option defines the maximum size of the internal buffer. Think of it as the capacity of your waiting room:

// Small buffer - fills up quickly
const smallBuffer = fs.createWriteStream("output.txt", {
  highWaterMark: 1024, // 1 KB
});

// Large buffer - can hold more data before filling
const largeBuffer = fs.createWriteStream("output.txt", {
  highWaterMark: 65536, // 64 KB
});

// Default buffer - balanced for most use cases
const defaultBuffer = fs.createWriteStream("output.txt");
// Default highWaterMark is 16384 (16 KB)

Here's the critical insight: When the buffer reaches its highWaterMark, Node.js needs to signal: "Stop sending data! I need time to process what I have."

This signal is called backpressure.

---

Backpressure: The Stream's "Slow Down" Signal

Let's discover what backpressure is and why it's one of the most important concepts in stream processing.

What Is Backpressure?

Backpressure is a signal that flows backward from the writable stream to the readable stream, saying: "I'm overwhelmed. Please pause."

Think of it like a traffic light:

• 🟢 Green light (true): "Keep writing! I can handle more data"
• 🔴 Red light (false): "Stop! My buffer is full, wait for me to catch up"

Here's how Node.js implements this signal:

const canContinue = writeStream.write(data);

if (canContinue === true) {
  // 🟢 Green light: Buffer has space, keep going!
} else {
  // 🔴 Red light: Buffer is full, need to pause!
}

The Boolean Return Value of write()

Every time you call write(), it returns a boolean value:

// Returns true: Data successfully buffered, space still available
const result1 = writeStream.write("Hello");
console.log(result1); // true - plenty of space

// Returns false: Data buffered, but buffer is now full
const result2 = writeStream.write(largeData);
console.log(result2); // false - buffer reached highWaterMark

Important to understand:

• When write() returns false, the data is still accepted and buffered
• But it's a warning: "Don't send more data right now!"
• If you ignore this warning and keep writing, the buffer grows beyond highWaterMark
• This can lead to memory overflow and crashes

What Happens If You Ignore Backpressure?

Let me show you what I discovered the hard way:

// ❌ Ignoring backpressure - DANGEROUS
const readStream = fs.createReadStream("huge-file.mp4");
const writeStream = fs.createWriteStream("copy.mp4");

readStream.on("data", (chunk) => {
  // Just write, don't check the return value
  writeStream.write(chunk);

  // Problem: If writeStream can't keep up,
  // chunks pile up in memory
  // Eventually: OUT OF MEMORY crash!
});

What happens behind the scenes:

Time: 0s
Memory: 50 MB [Normal]
Buffer: ████░░░░░░ (40% full)

Time: 5s
Memory: 200 MB [Getting high]
Buffer: ████████░░ (80% full)
- writeStream.write() returns false
- But we ignore it and keep writing!

Time: 10s
Memory: 800 MB [Danger zone]
Buffer: ██████████████████ (160% full - exceeding limit!)
- Buffer growing beyond highWaterMark
- Memory usage climbing

Time: 15s
Memory: 2 GB [Critical]
Process: CRASH - Out of memory

This is why backpressure exists: to prevent exactly this scenario.

---

The Relationship Between Readable and Writable Streams

Now that we understand backpressure, let's discover how readable and writable streams work together.

Both Sides Have Buffers

Here's something important to understand: both readable and writable streams have their own internal buffers:

const readStream = fs.createReadStream("input.txt", {
  highWaterMark: 64 * 1024, // 64 KB read buffer
});

const writeStream = fs.createWriteStream("output.txt", {
  highWaterMark: 16 * 1024, // 16 KB write buffer
});

The data flow looks like this:

[Source File]
     ↓
[Read Buffer: 64 KB] ← Fills from file
     ↓
[Your Code: stream.on('data')]
     ↓
[Write Buffer: 16 KB] ← Receives data from read stream
     ↓
[Destination File]

Why Reading Is Usually Faster

Writable streams are generally slower because writing involves more complex operations:

Reading operations:

1. Load data from disk into memory
2. Pass chunk to your code

Writing operations:

1. Receive chunk from your code
2. Apply encoding (UTF-8, ASCII, etc.)
3. Format data if needed
4. Wait for disk/network to be ready
5. Actually write the bytes
6. Verify write succeeded
7. Update file descriptor position

See the difference? Writing has more steps, which makes it slower.

The Speed Mismatch in Action

Let's see this with actual numbers:

import fs from "fs";

const readStream = fs.createReadStream("large-file.txt");
const writeStream = fs.createWriteStream("output.txt");

let readChunks = 0;
let writeChunks = 0;

readStream.on("data", (chunk) => {
  readChunks++;
  console.log(`Read chunk ${readChunks} (size: ${chunk.length} bytes)`);

  const canWrite = writeStream.write(chunk);
  writeChunks++;
  console.log(`Wrote chunk ${writeChunks} - Can continue: ${canWrite}`);
});

Output might look like:

Read chunk 1 (size: 65536 bytes)
Wrote chunk 1 - Can continue: true
Read chunk 2 (size: 65536 bytes)
Wrote chunk 2 - Can continue: true
Read chunk 3 (size: 65536 bytes)
Wrote chunk 3 - Can continue: false  ← Buffer full!
Read chunk 4 (size: 65536 bytes)
Wrote chunk 4 - Can continue: false  ← Still full!

Notice how after chunk 3, the write buffer fills up? The stream signals backpressure with false, but the read stream keeps sending data because we haven't told it to pause.

This is the problem we need to solve.

---

Comparing: With and Without Backpressure Handling

Let's see the dramatic difference between handling backpressure properly and ignoring it. I'll show you both approaches with a real example: copying a large file.

Without Backpressure Handling: The Dangerous Way

import fs from "fs";
import path from "path";

const sourceFile = path.join(__dirname, "large_input.txt");
const destFile = path.join(__dirname, "output_no_backpressure.txt");

// Create a test file if it doesn't exist (100,000 lines)
if (!fs.existsSync(sourceFile)) {
  const data = "This is sample data that will be repeated many times.\n".repeat(
    100000
  );
  fs.writeFileSync(sourceFile, data);
  console.log("Created test file:", sourceFile);
}

const readStream = fs.createReadStream(sourceFile);
const writeStream = fs.createWriteStream(destFile);

console.log("Copying started (without backpressure handling)...");

// ❌ DANGEROUS: Ignoring backpressure
readStream.on("data", (chunk) => {
  // Just write without checking if we can continue
  writeStream.write(chunk);

  // Problem: No pause/resume logic
  // If write buffer fills up, chunks pile up in memory
});

readStream.on("end", () => {
  writeStream.end();
  console.log("Copy complete (without backpressure handling).");
});

readStream.on("error", (err) => console.error("Read error:", err));
writeStream.on("error", (err) => console.error("Write error:", err));

What happens behind the scenes:

Memory Usage Over Time (Without Backpressure):

Start:    50 MB  ████░░░░░░░░░░░░░░░░
5 sec:   150 MB  ████████░░░░░░░░░░░░
10 sec:  400 MB  ████████████████░░░░
15 sec:  800 MB  ████████████████████ ⚠️ High!
20 sec: 1.5 GB  ████████████████████ ❌ Crash risk!

With Backpressure Handling: The Safe Way

Now let's see the correct approach:

import fs from "fs";
import path from "path";

const sourceFile = path.join(__dirname, "large_input.txt");
const destFile = path.join(__dirname, "output_with_backpressure.txt");

const readStream = fs.createReadStream(sourceFile);
const writeStream = fs.createWriteStream(destFile);

console.log("Copying started (with backpressure handling)...");

// ✅ SAFE: Handling backpressure properly
readStream.on("data", (chunk) => {
  // Check if we can continue writing
  const canContinue = writeStream.write(chunk);

  if (!canContinue) {
    // Buffer is full! Pause reading temporarily
    readStream.pause();
    console.log("⏸️  Backpressure detected — pausing read stream...");
  }
});

// ✅ Resume reading when write buffer drains
writeStream.on("drain", () => {
  console.log("✅ Drain event — write buffer cleared, resuming read stream...");
  readStream.resume();
});

readStream.on("end", () => {
  writeStream.end();
  console.log("Copy complete (with backpressure handling).");
});

readStream.on("error", (err) => console.error("Read error:", err));
writeStream.on("error", (err) => console.error("Write error:", err));

What happens behind the scenes:

Memory Usage Over Time (With Backpressure):

Start:    50 MB  ████░░░░░░░░░░░░░░░░
5 sec:    65 MB  █████░░░░░░░░░░░░░░░
10 sec:   68 MB  █████░░░░░░░░░░░░░░░ ✅ Stable
15 sec:   67 MB  █████░░░░░░░░░░░░░░░ ✅ Stable
20 sec:   66 MB  █████░░░░░░░░░░░░░░░ ✅ Stable
Complete: 50 MB  ████░░░░░░░░░░░░░░░░

See the difference? With proper backpressure handling, memory usage stays stable!

Understanding the Drain Event

The 'drain' event is crucial to backpressure handling. Let's understand when it fires:

writeStream.on("drain", () => {
  console.log("Drain event fired!");
  // This means: "I've finished processing buffered data"
  // "You can safely write more now"
});

The drain event fires when:

1. The write buffer was full (write() returned false)
2. The stream has processed enough data to free up space
3. The buffer is now below highWaterMark again

Visual timeline:

Time: 0s
Buffer: [░░░░░░░░░░] (Empty)
write('data1') → returns true

Time: 1s
Buffer: [████░░░░░░] (40% full)
write('data2') → returns true

Time: 2s
Buffer: [██████████] (100% full - at highWaterMark)
write('data3') → returns false ⚠️

Time: 3s
[Stream processes data, buffer empties]
Buffer: [████░░░░░░] (40% full)
🔔 'drain' event fires!

Time: 4s
Can safely write again! ✅

Side-by-Side Comparison

Let's put it all together in a comparison table:

Feature	Without Backpressure	With Backpressure
Memory Usage	Grows uncontrollably	Stays stable and controlled
Performance	Fast initially, then crashes	Consistent and reliable
Risk Level	❌ High - can crash production	✅ Safe - production-ready
Code Complexity	Simple but dangerous	Slightly more code, much safer
When to Use	Never in production	Always for large data

---

Common Misconceptions

Let's address some misunderstandings I had when learning about backpressure.

❌ Misconception: "write() returning false means the data wasn't written"

Reality: When write() returns false, the data is still accepted and placed in the buffer. The false is just a warning signal.

Why this matters: You don't need to retry the write or store the data elsewhere. It's already buffered.

Example:

const canContinue = writeStream.write("Hello");
console.log(canContinue); // false

// ❌ Wrong: Trying to write again
if (!canContinue) {
  writeStream.write("Hello"); // Don't do this! Data already buffered
}

// ✅ Correct: Just pause and wait
if (!canContinue) {
  readStream.pause(); // Wait for 'drain' event
}

❌ Misconception: "Small files don't need backpressure handling"

Reality: Even small files can cause issues if you process many of them concurrently, or if the destination is slow (network, remote disk).

Why this matters: Always handle backpressure unless you're absolutely certain about your data size and processing speed.

Example:

// Processing 1000 small files concurrently
for (let i = 0; i < 1000; i++) {
  processFile(`file${i}.txt`);
  // Even if each file is small, 1000 at once can overflow memory
}

❌ Misconception: "Backpressure only matters for file streams"

Reality: Backpressure applies to any writable stream - network sockets, HTTP responses, database connections, custom streams.

Why this matters: You'll encounter writable streams in many contexts beyond files.

Example:

// HTTP response is a writable stream
app.get("/download", (req, res) => {
  const fileStream = fs.createReadStream("large-file.zip");

  // ❌ Wrong: No backpressure handling
  fileStream.on("data", (chunk) => {
    res.write(chunk); // Can overflow if client is slow
  });

  // ✅ Correct: Use pipe (handles backpressure automatically)
  fileStream.pipe(res);
});

❌ Misconception: "The buffer can't grow beyond highWaterMark"

Reality: The buffer can grow beyond highWaterMark if you ignore backpressure signals. highWaterMark is a threshold, not a hard limit.

Why this matters: Ignoring backpressure can still cause memory issues even with a highWaterMark set.

Example:

const writeStream = fs.createWriteStream("output.txt", {
  highWaterMark: 1024, // 1 KB limit
});

// If you ignore the false return value:
for (let i = 0; i < 1000; i++) {
  writeStream.write("x".repeat(1024)); // Each write is 1 KB
  // Even if write() returns false, we keep writing
  // Buffer grows to 1000 KB (1 MB) in memory!
}

console.log(writeStream.writableLength); // Could be much larger than 1024!

---

Troubleshooting Common Issues

Problem: Application crashes with "Out of Memory" error

Symptoms:

• Application runs fine with small files
• Crashes when processing large files
• Memory usage grows continuously
• Error: "JavaScript heap out of memory"

Common Causes:

1. Not handling backpressure (70% of cases) - Ignoring the boolean return value of write()
2. No drain event listener (20% of cases) - Not resuming after pause
3. Processing multiple large streams concurrently (10% of cases) - Too many streams at once

Diagnostic Steps:

// Step 1: Add memory monitoring
readStream.on("data", (chunk) => {
  console.log(
    "Memory usage:",
    process.memoryUsage().heapUsed / 1024 / 1024,
    "MB"
  );

  // Step 2: Check if backpressure is being signaled
  const canContinue = writeStream.write(chunk);
  console.log("Can continue writing:", canContinue);

  if (!canContinue) {
    console.log("⚠️ Backpressure detected - are we pausing?");
  }
});

Solution:

// Add proper backpressure handling
readStream.on("data", (chunk) => {
  const canContinue = writeStream.write(chunk);

  if (!canContinue) {
    readStream.pause(); // Pause reading
  }
});

// Add drain event handler
writeStream.on("drain", () => {
  readStream.resume(); // Resume reading
});

Prevention:

• Always check the return value of write()
• Always implement drain event handling
• Test with large files during development

Problem: Stream never completes or hangs indefinitely

Symptoms:

• File copying starts but never finishes
• No error messages
• Process doesn't exit
• CPU usage stays low

Common Causes:

1. Paused stream never resumed - Forgot to handle drain event
2. Missing end() call - Didn't close the writable stream
3. Event listener registered multiple times - Causes resume to be called multiple times

Diagnostic Steps:

// Add logging to track stream state
readStream.on("data", (chunk) => {
  console.log("📖 Read chunk, isPaused:", readStream.isPaused());

  const canContinue = writeStream.write(chunk);
  if (!canContinue) {
    console.log("⏸️  Pausing...");
    readStream.pause();
  }
});

writeStream.on("drain", () => {
  console.log("✅ Drain event, isPaused:", readStream.isPaused());
  readStream.resume();
});

readStream.on("end", () => {
  console.log("📗 Read stream ended");
  writeStream.end();
});

writeStream.on("finish", () => {
  console.log("✍️  Write stream finished");
});

Solution:

// Make sure drain handler is registered ONCE
writeStream.once("drain", () => {
  // Use 'once' to prevent multiple registrations
  readStream.resume();
});

// Make sure to call end()
readStream.on("end", () => {
  writeStream.end(); // This is critical!
});

Prevention:

• Always register drain listener
• Always call writeStream.end() when done
• Use once() instead of on() if appropriate

Problem: Data appears corrupted or incomplete in output file

Symptoms:

• Output file size doesn't match input
• Random bytes missing or incorrect
• File format errors when opening

Common Causes:

1. Called end() too early - Before all data was written
2. Error during write ignored - Write failed silently
3. Encoding mismatch - Wrong character encoding

Diagnostic Steps:

// Track bytes written
let totalBytesRead = 0;
let totalBytesWritten = 0;

readStream.on("data", (chunk) => {
  totalBytesRead += chunk.length;
  console.log("Total read:", totalBytesRead);

  writeStream.write(chunk);
});

writeStream.on("finish", () => {
  console.log("Total bytes read:", totalBytesRead);
  console.log("Total bytes written:", totalBytesWritten);

  // Compare with original file size
  const stats = fs.statSync(sourceFile);
  console.log("Original file size:", stats.size);
});

Solution:

// Always wait for 'end' before calling end()
readStream.on("end", () => {
  console.log("All data read, now ending write stream");
  writeStream.end(); // Call after all data is read
});

// Handle errors
writeStream.on("error", (err) => {
  console.error("Write error:", err);
  readStream.destroy(); // Stop reading on error
});

Prevention:

• Only call end() after all data is read
• Always handle error events
• Verify file sizes match after copying

---

Check Your Understanding

Let's test what you've learned about writable streams and backpressure.

Question 1: What does backpressure prevent?

Show Answer

Answer: Backpressure prevents memory overflow by signaling when data is arriving faster than it can be processed.

Explanation: When a writable stream's buffer fills up, it returns false from write() to signal: "Slow down! I can't process data this fast." This prevents the application from buffering unlimited data in memory, which would eventually cause an out-of-memory crash.

Without backpressure handling, data accumulates in memory until the process crashes.

Question 2: What's wrong with this code?

const readStream = fs.createReadStream("large-file.txt");
const writeStream = fs.createWriteStream("output.txt");

readStream.on("data", (chunk) => {
  const canContinue = writeStream.write(chunk);

  if (!canContinue) {
    readStream.pause();
  }
});

Show Answer

Answer: Missing the drain event listener to resume the read stream!

Explanation: While this code correctly pauses the read stream when backpressure is detected, it never resumes. The read stream will pause and stay paused forever.

Fixed version:

const readStream = fs.createReadStream("large-file.txt");
const writeStream = fs.createWriteStream("output.txt");

readStream.on("data", (chunk) => {
  const canContinue = writeStream.write(chunk);

  if (!canContinue) {
    readStream.pause();
  }
});

// ✅ Add this!
writeStream.on("drain", () => {
  readStream.resume(); // Resume when buffer is drained
});

Question 3: When does write() return false?

Show Answer

Answer: write() returns false when the internal buffer reaches or exceeds its highWaterMark threshold.

Explanation: The highWaterMark (default 16 KB for file streams) defines when the buffer is "full enough" to signal backpressure. Once the buffered data reaches this threshold, write() returns false to indicate: "I'm still accepting your data, but please pause - I need time to process what I have."

Important: The data is still written to the buffer even when write() returns false. The return value is just a signal, not a rejection.

Example:

const writeStream = fs.createWriteStream("output.txt", {
  highWaterMark: 5, // 5 bytes
});

writeStream.write("abcd"); // 4 bytes, returns true
writeStream.write("e"); // 5 bytes total, returns false
// Buffer is at highWaterMark, signaling backpressure

Question 4: Why is writing usually slower than reading?

Show Answer

Answer: Writing involves more complex operations like encoding, formatting, waiting for I/O, and verification, while reading primarily just loads data into memory.

Explanation:

Reading operations (simpler):

1. Load data from disk into memory
2. Pass chunk to application

Writing operations (more complex):

1. Receive data from application
2. Apply character encoding (UTF-8, etc.)
3. Format/transform data if needed
4. Wait for disk/network to be ready
5. Write bytes to destination
6. Verify write succeeded
7. Update file position/metadata

This is why writable streams often become the bottleneck in data processing, and why backpressure management is so important.

Hands-On Challenge

Challenge: Create a function that copies a file with proper backpressure handling and logs progress.

Requirements:

• Accept source and destination file paths
• Handle backpressure correctly
• Log when pausing and resuming
• Track total bytes copied
• Handle errors gracefully

Starter Code:

import fs from "fs";

function copyFileWithBackpressure(source: string, destination: string) {
  // Your implementation here
}

// Test it
copyFileWithBackpressure("input.txt", "output.txt");

Show Solution

import fs from "fs";

function copyFileWithBackpressure(
  source: string,
  destination: string
): Promise<void> {
  return new Promise((resolve, reject) => {
    // Create streams
    const readStream = fs.createReadStream(source);
    const writeStream = fs.createWriteStream(destination);

    // Track progress
    let totalBytes = 0;
    let pauseCount = 0;

    // Handle data with backpressure
    readStream.on("data", (chunk: Buffer) => {
      totalBytes += chunk.length;

      // Write and check if we can continue
      const canContinue = writeStream.write(chunk);

      if (!canContinue) {
        // Backpressure detected
        pauseCount++;
        console.log(`⏸️  Paused at ${totalBytes} bytes (pause #${pauseCount})`);
        readStream.pause();
      }
    });

    // Resume when buffer drains
    writeStream.on("drain", () => {
      console.log(`✅ Resumed after drain event`);
      readStream.resume();
    });

    // Complete successfully
    readStream.on("end", () => {
      writeStream.end();
    });

    writeStream.on("finish", () => {
      console.log(
        `✅ Copy complete! Total: ${totalBytes} bytes, Paused: ${pauseCount} times`
      );
      resolve();
    });

    // Handle errors
    readStream.on("error", (err) => {
      console.error("Read error:", err);
      writeStream.destroy();
      reject(err);
    });

    writeStream.on("error", (err) => {
      console.error("Write error:", err);
      readStream.destroy();
      reject(err);
    });
  });
}

// Test it
copyFileWithBackpressure("large-input.txt", "output.txt")
  .then(() => console.log("Success!"))
  .catch((err) => console.error("Failed:", err));

Why this solution works:

1. Proper backpressure: Checks write() return value and pauses when needed
2. Resume mechanism: Listens to drain event to resume reading
3. Progress tracking: Logs bytes copied and pause events
4. Error handling: Handles errors from both streams and cleans up
5. Promise-based: Returns a promise for easy async/await usage
6. Resource cleanup: Destroys streams on errors to prevent leaks

Example output:

⏸️  Paused at 65536 bytes (pause #1)
✅ Resumed after drain event
⏸️  Paused at 131072 bytes (pause #2)
✅ Resumed after drain event
⏸️  Paused at 196608 bytes (pause #3)
✅ Resumed after drain event
✅ Copy complete! Total: 250000 bytes, Paused: 3 times
Success!

---

Summary: Key Takeaways

Let's recap what we've discovered on our journey through writable streams and backpressure:

Core Concepts:

• 🎯 Writable streams have internal buffers that temporarily store data waiting to be processed
• 🎯 highWaterMark defines buffer capacity - typically 16 KB for file streams
• 🎯 Backpressure signals "slow down" when the buffer reaches its limit
• 🎯 write() returns a boolean - false means buffer is full, true means continue
• 🎯 Reading is usually faster than writing due to complex write operations

Critical Implementation Points:

• ✅ Always check the return value of write()
• ✅ Pause the readable stream when write() returns false
• ✅ Listen for the drain event to know when to resume
• ✅ Call end() on the writable stream when done
• ✅ Handle errors on both readable and writable streams

The Backpressure Pattern:

readStream.on("data", (chunk) => {
  const canContinue = writeStream.write(chunk);
  if (!canContinue) readStream.pause();
});

writeStream.on("drain", () => {
  readStream.resume();
});

Why This Matters: Without proper backpressure handling, your application can:

• ❌ Crash with out-of-memory errors
• ❌ Consume excessive resources
• ❌ Become slow and unresponsive
• ❌ Lose or corrupt data

With proper backpressure handling:

• ✅ Stable memory usage regardless of file size
• ✅ Reliable performance in production
• ✅ Predictable resource consumption
• ✅ Safe data processing

---

Version Information

Tested with:

• Node.js: v18.x, v20.x, v22.x
• TypeScript: v5.x
• Works with both CommonJS and ES Modules

Known Compatibility Notes:

• Stream APIs are stable since Node.js v10
• Backpressure mechanism unchanged since early Node.js versions
• Works identically across all operating systems (Windows, macOS, Linux)

Future-Proof:

• These concepts are fundamental to Node.js and unlikely to change
• Newer Node.js versions may add features but won't break these patterns

---

Additional Resources

Official Documentation:

• Node.js Stream API
• Stream Implementers Guide

Community Resources:

• Stream Handbook - Comprehensive guide to Node.js streams
• Backpressure Explained - Official Node.js guide

Tools for Debugging Streams:

• why-is-node-running - Debug hanging streams
• Node.js Inspector - Built-in debugging