JavaScript Web Workers logo JS Web Workers

Processing Large Files with Blob.slice in Workers

file.arrayBuffer() on a 4 GB upload is a request for 4 GB of contiguous heap. It fails on most phones, fails on many laptops, and on the machines where it succeeds it costs a multi-second stall. slice() is the alternative, and it is free.

This guide is the file-input case of Streaming & Backpressure Across Threads, part of High-Performance Computation Patterns. Slicing gives you explicit control over how much of a file is resident at once, plus the ability to hand different ranges to different workers — something a single sequential stream cannot do.

The Bounded Read Loop

// hash.worker.ts — read a file in 8 MB slices, never more than one resident
const SLICE = 8 * 1024 * 1024;

self.addEventListener('message', async (event: MessageEvent<{ file: File }>) => {
  const { file } = event.data;                     // File objects are cloneable
  const stamp = file.lastModified;                 // detect an edit mid-read
  const digest = createRollingDigest();
  let offset = 0;

  try {
    while (offset < file.size) {
      const end = Math.min(offset + SLICE, file.size);
      // slice() is a descriptor — nothing is read until arrayBuffer() resolves.
      const bytes = new Uint8Array(await file.slice(offset, end).arrayBuffer());
      digest.update(bytes);
      offset = end;
      self.postMessage({ kind: 'PROGRESS', done: offset, total: file.size });
    }
    if (file.lastModified !== stamp) throw new Error('file changed during read');
    self.postMessage({ kind: 'DONE', digest: digest.hex() });
  } catch (error) {
    self.postMessage({ kind: 'ERROR', message: (error as Error).message });
  }
});

Peak memory here is one slice — 8 MB — regardless of whether the file is 40 MB or 40 GB. The loop is sequential by construction: the next slice is not requested until the previous one has been processed, which is backpressure obtained by not asking for more.

Resident memory reading a 4 GB file three ways Three approaches compared. Reading the whole file with arrayBuffer requires four gigabytes of contiguous heap, which fails outright on typical mobile devices and stalls for seconds where it succeeds. Reading in eight megabyte slices sequentially holds one slice at a time, so peak memory is eight megabytes at any file size. Reading four slices concurrently across a worker pool holds four slices at once for thirty-two megabytes, trading a small amount of memory for parallel throughput. Peak resident bytes for the same 4 GB file arrayBuffer() 4 GB contiguous — allocation failure on most phones 8 MB slices 8 MB — constant at any file size 4 × 8 MB in a pool 32 MB — four slices resident, four cores busy 0 2 GB 4 GB The olive and mustard bars are drawn to scale — at this axis they are almost invisible, which is the point.
Slice size sets the memory ceiling directly, so it is a number you choose rather than a consequence of the input you were given.

Line-by-Line

File objects survive postMessage. A File (and any Blob) is structured-cloneable, and cloning one does not copy its contents — the clone references the same underlying data. Passing the file to the worker and slicing there is therefore the correct division: the page never allocates a byte.

slice(start, end) allocates nothing. It returns a Blob describing a range. Creating every slice up front to build a work list is entirely reasonable; the cost arrives only at arrayBuffer().

await slice.arrayBuffer() is the read. This is where disk I/O happens and where the memory appears. It is also the only line that can fail with a NotReadableError, and the failure is asynchronous, so it must be caught rather than guarded.

file.lastModified bracketing. A File is a reference to a path, not a lock. Checking the timestamp before and after a long read turns “silently produced a hash of half the old file and half the new one” into a clean error.

Progress per slice, not per record. One message per 8 MB slice is 500 messages for a 4 GB file — smooth for a progress bar and negligible for the transport. Reporting per record would be millions of hops.

Records That Straddle Slice Boundaries

Byte ranges do not respect record boundaries: a slice ends mid-line, mid-JSON-object, mid-UTF-8-codepoint. Three techniques cover essentially every format.

Carry the tail. For newline-delimited data, keep the bytes after the last delimiter and prepend them to the next slice. Decode with a streaming decoder so a multi-byte character split across the boundary is handled for you:

const decoder = new TextDecoder('utf-8');
let tail = '';

for (const [start, end] of ranges) {
  const buf = await file.slice(start, end).arrayBuffer();
  // `stream: true` keeps a partial code point for the next call.
  const text = tail + decoder.decode(new Uint8Array(buf), { stream: true });
  const lines = text.split('\n');
  tail = lines.pop() ?? '';
  for (const line of lines) handle(line);
}
if (tail) handle(tail);

Overlap and discard. For fixed-maximum-length records, read SLICE + maxRecordLength bytes and stop at the last complete record; the next slice starts where the last complete record ended. Costs a little duplicate I/O and needs no carried state, which makes it the right choice when slices are processed in parallel.

Align to the format. Fixed-width records, or containers with an index — Parquet, many binary logs — let you compute boundaries exactly. Where the format allows it this is strictly best: no overlap, no carried state, and slices are independent, which is what makes a pool usable.

Three ways to handle a record split by a slice boundary Three strips of the same file. In the carry-the-tail strategy, the bytes after the last complete record in slice one are held and prepended to slice two, which requires sequential processing. In the overlap strategy, each slice reads a little past its end so it always contains one complete record beyond the boundary, and the duplicate region is discarded, allowing slices to be processed independently. In the aligned strategy, boundaries are computed from the format itself so no record is ever split and no overlap or carried state is needed. A record split by a byte boundary, three ways carry the tail slice 1 slice 2 the partial record is held and prepended — cheap, but slices must run in order overlap slice 1 + tail bytes slice 2 + tail bytes a little duplicate I/O, zero shared state — slices can be processed in any order aligned ends on a record starts on a record boundaries computed from the format — best when the format permits it Only the middle and bottom strips can be handed to a pool; the top one is inherently sequential.
Choosing a boundary strategy is really choosing whether the file can be processed in parallel — which is why it is worth deciding before writing the parser.

Parallel Slices Across a Pool

Once slices are independent, the file becomes an embarrassingly parallel workload. The dispatcher hands ranges — not bytes — to each worker, and each worker reads its own range:

// page.ts — the file object is cloned to each worker; no bytes are copied
const RANGES = 8 * 1024 * 1024;
const ranges: Array<[number, number]> = [];
for (let o = 0; o < file.size; o += RANGES) ranges.push([o, Math.min(o + RANGES, file.size)]);

const results = await pool.mapOrdered(ranges, (worker, [start, end]) =>
  worker.request({ kind: 'SCAN', file, start, end }));

Two constraints decide whether this pays. Disk throughput is shared, so on a spinning disk or a slow SD card four readers are not four times faster and may be slower due to seeking; on an SSD they usually are. And the aggregation must be associative — counts, sums, min/max, histograms and most digests combine fine; anything that depends on order needs the results reassembled by range index first.

Concurrency also multiplies the memory ceiling: poolSize × sliceSize. Four workers with 8 MB slices is 32 MB, which is fine; eight workers with 64 MB slices is half a gigabyte, which is not. Size the slice after choosing the pool size, following Worker Pool Management.

File System Access and the Origin Private File System

Two newer APIs change the arithmetic for repeat processing, and both are worker-friendly.

FileSystemFileHandle (from showOpenFilePicker()) is a durable reference the page can keep across sessions, and it is structured-cloneable, so it can be sent to a worker. Calling await handle.getFile() produces a fresh File snapshot, which is the clean answer to the stale-handle problem: re-snapshot before a long job instead of holding a File for minutes.

The origin private file system goes further. Inside a worker — and only inside a worker — createSyncAccessHandle() gives synchronous, seekable reads and writes over a file the origin owns:

// opfs.worker.ts — synchronous random access, worker-only by design
const root = await navigator.storage.getDirectory();
const handle = await root.getFileHandle('scratch.bin', { create: true });
const access = await handle.createSyncAccessHandle();

const view = new Uint8Array(8 * 1024 * 1024);
const read = access.read(view, { at: offset });     // no promise, no copy through a Blob
access.close();

That synchronous shape is why the API is restricted to workers: it would block the main thread outright. For workloads that pass over the same data repeatedly — building an index, then querying it — writing a normalised copy into the origin private file system once and reading it back synchronously is typically several times faster than re-slicing the user’s original file each pass, and it survives a page reload.

The trade is storage quota and the need to invalidate the derived copy when the source changes. Both are ordinary cache concerns; neither changes the slicing strategy above, which still applies to the first pass over the user’s file.

How each large-file strategy fails, and on what Four strategies with their failure modes. Reading the whole file into an ArrayBuffer fails to allocate on typical mobile devices. Reading it as text allocates twice the file size as a string before parsing starts. Sequential slices hold one slice and cannot use more than one core. Parallel slices multiply the memory ceiling by the pool size and require the work to partition cleanly. The failure mode of each approach file.arrayBuffer() — the whole file at once allocation failure on mobile; multi-second stall where it succeeds file.text() — the whole file as a string roughly twice the file size in memory before parsing begins sequential slices — one resident at a time safe and single-core; throughput is bounded by one thread parallel slices across a pool memory ceiling multiplies by pool size; needs independent ranges Only the lower two have a ceiling you choose rather than discover.
The first two rows are the same bug at different sizes: peak memory proportional to the input rather than to the design.

Gotchas

FileReader instead of arrayBuffer(). The callback-based FileReader still works and is strictly worse: it needs an event dance, it is easy to leak, and its onload closure captures the buffer. blob.arrayBuffer() is promise-based and available in workers everywhere.

Slices of slices are cheap, but the arithmetic is relative. blob.slice(a, b).slice(c, d) addresses c–d within the first slice. Convenient for framing a record, and a source of off-by-megabyte errors when the two coordinate systems get mixed.

Mobile memory ceilings are lower than they look. iOS terminates a tab that allocates too aggressively without a catchable error. Keeping slices at 4–8 MB and concurrency at two on mobile avoids the class of crash entirely.

Holding every slice’s result until the end. Bounding the input is pointless if the outputs accumulate: a scan that keeps one parsed object per row ends up with the whole file on the heap in a different shape. Aggregate incrementally, or stream the results onward under their own flow control.

Progress that reports read bytes, not processed bytes. With a pool, slices complete out of order, so a progress bar driven by “highest offset seen” jumps and stalls. Track the sum of completed slice sizes instead.

The file picker’s File can outlive its validity. A handle retained in application state for minutes may reference a file the user has since replaced. Re-check lastModified before a long job, not only during it.

Performance Note

Hashing a 4 GB file on a 2023 laptop with an NVMe drive: a single worker with 8 MB slices sustained 1.4 GB/s and held 8 MB resident. Four workers over independent ranges reached 2.9 GB/s — sublinear, because the digest itself is CPU-bound and the drive is shared — at 32 MB resident. The whole-file arrayBuffer() approach could not be measured on the same machine: it failed to allocate.

Frequently Asked Questions

Does Blob.slice() copy the data?
No. slice() returns a new Blob that references a byte range of the original — it is a descriptor, not a buffer, and creating a thousand slices costs microseconds and no memory. The read is what costs: await slice.arrayBuffer() is the moment those bytes are pulled from disk into the heap, so memory is governed by how many slices you have resolved at once, not how many you created.
Why does my read throw NotReadableError halfway through a big file?
The underlying file changed or became unavailable after the File handle was obtained — the user edited or moved it, a sync client rewrote it, or an external drive was unmounted. A File is a snapshot of a path plus a last-modified time, not a lock, so long reads must handle this: catch the failure, compare file.lastModified with the value captured at the start, and ask the user to re-select rather than silently producing a truncated result.

See also

Related Guides

High Performance Computation Patterns Implementing Backpressure With Message Credits Open guide High Performance Computation Patterns Piping Readable Streams Into Workers Open guide