Node.js Streams: Processing Large Files Efficiently

On this page

Streams let you process large files without loading everything into memory. Essential for scalable Node.js applications.

Reading a File Stream

JavaScript9 lines
const fs = require('fs');

const readStream = fs.createReadStream('large-file.txt');

readStream.on('data', chunk => {
  console.log(`Received ${chunk.length} bytes`);
});

readStream.on('end', () => console.log('Done'));

Piping Streams

JavaScript5 lines
const zlib = require('zlib');

fs.createReadStream('input.txt')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('output.txt.gz'));

Transform Stream

JavaScript10 lines
const { Transform } = require('stream');

const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

process.stdin.pipe(upperCase).pipe(process.stdout);

References


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.