# Buffers, Streams, and the File System in Nodejs

We'll walk through **Buffers**, the **File System (fs)**, and **Streams**, not as isolated APIs, but as ideas that work together. I'll keep the language simple, examples practical, and explanations grounded in how real systems behave.

## **Why This Matters**

Node.js is different from traditional backend runtimes because:

1. **It is single-threaded**
    
2. **It is event-driven**
    
3. **It is built for I/O**
    

Most Node apps spend their time:

* Reading files
    
* Writing files
    
* Sending data over the network
    
* Receiving data from the network
    

**Buffers, fs, and Streams** are Node's core tools for moving data efficiently.

## **Part 1: Buffers**

### **What is a Buffer?**

A Buffer is Node.js's way of handling **binary data**.

Think of it like: **"A box of raw bytes."**

This is important because:

* Files are bytes
    
* Network packets are bytes
    
* Images, videos, PDFs all are bytes
    

JavaScript itself wasn't designed for this kind of low-level data. So Node introduced `Buffer`.

```javascript

const buf = Buffer.from("hello");
console.log(buf);

// Output
<Buffer 68 65 6c 6c 6f>
```

**What's happening here?**

* `Buffer.from("hello")` takes the string "hello" and converts it into a sequence of bytes
    
* Each pair of characters (like `68`, `65`) represents one byte in hexadecimal format
    
* `68` in hex = 104 in decimal = the letter 'h' in ASCII/UTF-8
    
* `65` in hex = 101 in decimal = the letter 'e'
    
* And so on for `6c` (l), `6c` (l), `6f` (o)
    

```javascript
// Convert the buffer back to a human-readable string
console.log(buf.toString());
//Output 
"hello"
```

**What's happening here?**

* `buf.toString()` decodes those raw bytes back into a UTF-8 string
    
* By default, it assumes UTF-8 encoding
    
* You could specify other encodings like `buf.toString('base64')` or `buf.toString('hex')`
    

### **Buffers Are Fixed-Size**

```javascript

const buf = Buffer.alloc(5);
buf.write("hello world");

console.log(buf.toString());

// output 
"hello"
```

**What's happening here?**

* `Buffer.alloc(5)` allocates exactly 5 bytes of memory, initialized to zeros
    
* `buf.write("hello world")` attempts to write the full string
    
* But the buffer only has room for 5 bytes, so it only writes "hello"
    
* The rest (" world") is silently ignored
    
* This is not an error, it's by design
    

**I’m gonna tell you why?**

* Buffers have fixed memory
    
* They do not auto-expand
    
* This is intentional. It makes memory usage predictable and prevents runaway memory growth
    

## **Part 2: The File System (fs)**

Now let's talk about `fs`. At a high level the fs module lets Node talk to your hard disk**.** But the way you talk to the disk matters a lot.

### **Sync vs Async**

```javascript
// sync file operations
const fs = require("fs");
const data = fs.readFileSync("file.txt", "utf8");
console.log(data);
```

**What's happening here?**

* `require("fs")` loads Node's built-in file system module
    
* `fs.readFileSync("file.txt", "utf8")` tells the operating system to read "file.txt"
    
* The `Sync` suffix means **synchronous**, node waits right here
    
* While waiting, **nothing else runs**, the event loop is frozen
    
* Only when the OS finishes reading does execution continue
    
* `"utf8"` tells Node to decode the raw bytes into a UTF-8 string
    

This blocks the event loop.

```javascript
// async file operation
const fs = require("fs");

fs.readFile("file.txt", "utf8", (err, data) => {
  if (err) throw err;  
  console.log(data);  
});

console.log("This prints first!");
```

**What's happening here?**

* `fs.readFile()` is the async version, no `Sync` suffix
    
* Node asks the OS for the file, then **moves on immediately**
    
* The third argument is a **callback function** that runs later
    
* When the OS finishes reading, Node puts the callback in the event loop queue
    
* The callback receives two arguments:
    
    * `err`: an error object if something went wrong, otherwise `null`
        
    * `data`: the file contents as a string (because we specified `"utf8"`)
        

### **Under the Hood**

Node does not read the file itself. It:

1. Delegates to the OS
    
2. Registers a callback
    
3. Keeps the event loop free
    

This is why Node scales so well with I/O-heavy workloads.

### **But There's a Hidden Problem…**

Both `readFileSync` and `readFile` do something dangerous:

**They load the entire file into memory.**

That's fine for:

* Small files
    
* JSON configs
    
* Logs
    

But terrible for:

* Large files
    
* Videos
    
* Data pipelines
    

## **Part 3: Streams**

### **What is a Stream?**

A stream is data that:

* Arrives in chunks
    
* Over time
    
* Piece by piece
    

**Think:** Water flowing through a pipe, not a bucket dumped on your head.

---

### **Why Streams Exist**

Imagine reading a 5GB file.

**Option 1:** Load everything into memory ❌

**Option 2:** Read 64KB at a time, process it, move on ✅

**Streams are Option 2.**

---

### **Types of Streams in Nodejs**

There are four types, but we'll focus on three:

1. **Readable** - produces data
    
2. **Writable** - consumes data
    
3. **Transform** - modifies data in between
    

### Readable Streams

```javascript

const fs = require("fs");

const stream = fs.createReadStream("bigfile.txt", {
  encoding: "utf8", 
});


stream.on("data", chunk => {
  console.log("Received chunk:", chunk.length);
});

stream.on("end", () => {
  console.log("Done reading file");
});
```

**What's happening here?**

* `fs.createReadStream("bigfile.txt", {...})` creates a stream object
    
* Instead of reading the whole file, it opens it and prepares to read chunks
    
* The `encoding: "utf8"` option means chunks will be strings, not Buffers
    
* `stream.on("data", ...)` listens for the `data` event
    
* Every time the OS delivers a chunk (usually ~64KB), the callback runs
    
* `chunk` is the data, either a string (if encoding is set) or a Buffer
    
* `stream.on("end", ...)` listens for the `end` event, which fires when done
    

#### **Writable Streams**

Writable streams **consume** data.

```javascript

const fs = require("fs");

const writeStream = fs.createWriteStream("output.txt");
writeStream.write("Hello\n");

writeStream.write("World\n");

writeStream.end();
```

**What's happening here?**

* `fs.createWriteStream("output.txt")` creates a stream that writes to "output.txt"
    
* `writeStream.write("Hello\n")` puts data into the stream's internal buffer
    
* Node will flush this to disk asynchronously
    
* You can call `write()` multiple times
    
* `writeStream.end()` tells Node "no more data coming", it will finish writing and close the file
    

#### **Writable Streams**

Writable streams **consume** data.

```javascript

const fs = require("fs");

const writeStream = fs.createWriteStream("output.txt");
writeStream.write("Hello\n");
writeStream.write("World\n");
writeStream.end();
```

**What's happening here?**

* `fs.createWriteStream("output.txt")` creates a stream that writes to "output.txt"
    
* `writeStream.write("Hello\n")` puts data into the stream's internal buffer
    
* Node will flush this to disk asynchronously
    
* You can call `write()` multiple times
    
* `writeStream.end()` tells Node "no more data coming", it will finish writing and close the file
    
* Data flows into the stream.
    

mastering Buffers, the File System, and Streams in Node.js is essential for efficient data handling. Buffers manage binary data, crucial for files and network packets.