Streaming JSON Parsing with Transferable Chunks

Loading a 20 MB NDJSON file with await response.json() blocks the main thread for 40–80 ms while it buffers and parses the entire payload. Streaming it as transferable ArrayBuffer chunks into a worker cuts main-thread blocking to under 1 ms per chunk and lets users see progress.

This technique is part of the Data Parsing & Serialization section within the broader High-Performance Computation Patterns reference.

Minimal Reproducible Example

// main.ts — stream NDJSON from the network into a worker
const worker = new Worker(
  new URL('./ndjson.worker.ts', import.meta.url),
  { type: 'module' }
);

worker.onmessage = (e: MessageEvent) => {
  if (e.data.type === 'record') {
    appendRow(e.data.record);          // render each parsed record incrementally
  } else if (e.data.type === 'progress') {
    updateProgressBar(e.data.bytesReceived, e.data.totalBytes);
  } else if (e.data.type === 'done') {
    console.log(`Parsed ${e.data.count} records`);
    worker.terminate();
  }
};

async function streamToWorker(url: string): Promise<void> {
  const response = await fetch(url);
  const totalBytes = Number(response.headers.get('Content-Length') ?? 0);
  const reader = response.body!.getReader();
  let bytesReceived = 0;

  while (true) {
    const { done, value } = await reader.read();
    if (done) {
      worker.postMessage({ type: 'end' });
      break;
    }
    // value is a Uint8Array view over an ArrayBuffer
    bytesReceived += value.byteLength;
    const buffer = value.buffer.slice(
      value.byteOffset,
      value.byteOffset + value.byteLength
    );
    // Transfer ownership — zero copy, no main-thread heap growth
    worker.postMessage(
      { type: 'chunk', buffer, bytesReceived, totalBytes },
      [buffer]
    );
  }
}

// ndjson.worker.ts — decode and parse incrementally
const decoder = new TextDecoder('utf-8', { ignoreBOM: true });
let remainder = '';
let recordCount = 0;

self.onmessage = (e: MessageEvent) => {
  if (e.data.type === 'chunk') {
    const text = decoder.decode(e.data.buffer as ArrayBuffer, { stream: true });
    const lines = (remainder + text).split('\n');
    remainder = lines.pop() ?? '';     // last partial line carried over

    for (const line of lines) {
      const trimmed = line.trim();
      if (!trimmed) continue;
      try {
        const record = JSON.parse(trimmed);
        recordCount++;
        self.postMessage({ type: 'record', record });
      } catch {
        // Skip malformed lines
      }
    }

    self.postMessage({
      type: 'progress',
      bytesReceived: e.data.bytesReceived,
      totalBytes: e.data.totalBytes,
    });
  } else if (e.data.type === 'end') {
    // Flush the remaining partial line
    if (remainder.trim()) {
      try {
        self.postMessage({ type: 'record', record: JSON.parse(remainder) });
        recordCount++;
      } catch { /* ignore trailing malformed data */ }
    }
    self.postMessage({ type: 'done', count: recordCount });
    self.close();
  }
};

Line-by-Line Walkthrough

const { done, value } = await reader.read();

ReadableStreamDefaultReader.read() returns the next Uint8Array chunk as the network delivers it. Chunk sizes are controlled by the TCP receive buffer (typically 8–64 KB). This await yields to the event loop between chunks, keeping the main thread responsive throughout.

const buffer = value.buffer.slice(
  value.byteOffset,
  value.byteOffset + value.byteLength
);

A Uint8Array returned by the stream reader shares its underlying ArrayBuffer with the stream’s internal buffer. Slicing produces a new, detached ArrayBuffer covering only the chunk. This is necessary before transferring: you cannot transfer the shared buffer without detaching it from all other views, which would corrupt the stream reader.

worker.postMessage(
  { type: 'chunk', buffer, bytesReceived, totalBytes },
  [buffer]   // transfer list
);

Passing buffer in the transfer list hands ownership to the worker. After this call, buffer.byteLength === 0 on the main thread. The worker receives the full contents with no heap copy. For a 64 KB chunk this costs under 0.05 ms, versus ~0.1 ms for structured clone.

const decoder = new TextDecoder('utf-8', { ignoreBOM: true });
// …
const text = decoder.decode(e.data.buffer as ArrayBuffer, { stream: true });

TextDecoder with { stream: true } keeps internal state between calls. It correctly handles UTF-8 multi-byte sequences that straddle chunk boundaries — a byte sequence for é (0xC3 0xA9) split across two chunks is reassembled automatically. Without { stream: true }, the decoder treats each chunk as a complete string and replaces incomplete sequences with the replacement character (U+FFFD).

const lines = (remainder + text).split('\n');
remainder = lines.pop() ?? '';

Splitting on \n and storing the last element in remainder handles NDJSON records that span chunk boundaries. After pop(), lines contains only complete records. The last partial line accumulates in remainder until the next chunk fills it.

Gotchas & Edge Cases

1. Transferring a shared ArrayBuffer detaches all views

The slice step above is non-optional. If you skip it and transfer value.buffer directly, every other Uint8Array view over that buffer (inside the browser’s stream reader implementation) is simultaneously detached. This corrupts the stream and causes subsequent reader.read() calls to throw a TypeError.

2. Content-Length is absent on chunked-transfer-encoding responses

Servers using Transfer-Encoding: chunked do not send Content-Length. In that case response.headers.get('Content-Length') returns null. Guard against NaN progress fractions:

const totalBytes = Number(response.headers.get('Content-Length') ?? 0);
// In the worker, only show progress if totalBytes > 0
if (e.data.totalBytes > 0) {
  const pct = (e.data.bytesReceived / e.data.totalBytes) * 100;
  self.postMessage({ type: 'progress', pct });
}

3. Large record counts flood the message channel

Posting a message per record is practical up to ~50 000 records per second. Beyond that, postMessage overhead accumulates. Batch records into arrays of 100–500 and post the batch:

const BATCH_SIZE = 200;
const batch: unknown[] = [];

for (const line of lines) {
  const trimmed = line.trim();
  if (!trimmed) continue;
  try { batch.push(JSON.parse(trimmed)); } catch { /* skip */ }
  if (batch.length >= BATCH_SIZE) {
    self.postMessage({ type: 'batch', records: batch.splice(0) });
  }
}

4. Worker cleanup on fetch abort

If the user navigates away mid-stream, the ReadableStream must be cancelled and the worker terminated:

const controller = new AbortController();
const response = await fetch(url, { signal: controller.signal });

// On navigation / component unmount:
controller.abort();     // cancels the fetch and closes the ReadableStream
worker.terminate();     // terminates the worker, discarding in-flight chunks

Without explicit cancellation, the fetch and the worker continue running in the background, consuming both bandwidth and CPU.

Progress Reporting

The bytesReceived / totalBytes ratio is the simplest progress signal, but it underestimates work for large NDJSON files with variable-length records. An alternative is to count parsed records and report against an expected total known from a preceding metadata request:

// If the server provides a record count header:
const expectedCount = Number(response.headers.get('X-Record-Count') ?? 0);

// In the worker, after each batch:
self.postMessage({ type: 'progress', parsed: recordCount, total: expectedCount });

This produces a smoother progress bar because record parsing speed is more uniform than byte arrival rate (which fluctuates with TCP congestion).

Comparison: Streaming vs Buffered Approaches

The streaming approach is uniquely capable of showing the first record before the full download completes — critical for large paginated datasets where users need early feedback.

Handling Full JSON Arrays (Non-NDJSON)

NDJSON (one JSON object per line) is the ideal format for streaming because each newline is a record boundary. For a standard JSON array ([{…},{…},…]), you need a lightweight streaming JSON parser that understands bracket depth.

A minimal approach uses a depth counter to detect top-level object boundaries:

// array-stream.worker.ts — stream a top-level JSON array
const decoder = new TextDecoder('utf-8', { ignoreBOM: true });
let buffer = '';
let depth = 0;
let inString = false;
let escape = false;
let arrayStarted = false;
let recordCount = 0;

self.onmessage = (e: MessageEvent) => {
  if (e.data.type === 'end') {
    self.postMessage({ type: 'done', count: recordCount });
    self.close();
    return;
  }

  buffer += decoder.decode(e.data.buffer as ArrayBuffer, { stream: true });

  let start = 0;
  for (let i = 0; i < buffer.length; i++) {
    const ch = buffer[i];
    if (escape) { escape = false; continue; }
    if (ch === '\\' && inString) { escape = true; continue; }
    if (ch === '"') { inString = !inString; continue; }
    if (inString) continue;

    if (ch === '[' && !arrayStarted) { arrayStarted = true; start = i + 1; continue; }
    if (ch === '{') depth++;
    if (ch === '}') {
      depth--;
      if (depth === 0 && arrayStarted) {
        const json = buffer.slice(start, i + 1).trim();
        if (json) {
          try {
            self.postMessage({ type: 'record', record: JSON.parse(json) });
            recordCount++;
          } catch { /* skip */ }
        }
        start = i + 2; // skip comma after }
      }
    }
  }
  buffer = buffer.slice(start);
};

This handles arbitrarily large JSON arrays without loading the full document. The performance characteristic is similar to the NDJSON approach: 250 MB/s parsing throughput inside the worker, under 2 ms total main-thread blocking.

Integration with React and Framework Components

When using this pattern inside a React component, the worker lifecycle must be tied to the component lifecycle to prevent memory leaks on unmount:

// useNdjsonStream.ts
import { useEffect, useRef, useState } from 'react';

export function useNdjsonStream<T>(url: string) {
  const [records, setRecords] = useState<T[]>([]);
  const [progress, setProgress] = useState(0);
  const [done, setDone] = useState(false);
  const workerRef = useRef<Worker | null>(null);
  const controllerRef = useRef<AbortController | null>(null);

  useEffect(() => {
    const worker = new Worker(
      new URL('./ndjson.worker.ts', import.meta.url),
      { type: 'module' }
    );
    const controller = new AbortController();
    workerRef.current = worker;
    controllerRef.current = controller;

    worker.onmessage = (e: MessageEvent) => {
      if (e.data.type === 'record') {
        setRecords(prev => [...prev, e.data.record as T]);
      } else if (e.data.type === 'progress') {
        setProgress(e.data.bytesReceived / (e.data.totalBytes || 1));
      } else if (e.data.type === 'done') {
        setDone(true);
      }
    };

    streamToWorker(url /*, controller.signal */);

    return () => {
      controller.abort();
      worker.terminate();
    };
  }, [url]);

  return { records, progress, done };
}

The cleanup function returned from useEffect aborts the fetch and terminates the worker, preventing background processing after the component unmounts. Without the cleanup, the worker continues posting messages to a stale state setter after the component is gone.