How to Scrub Huge Video Files in the Browser Without Blowing Up RAM

Vidstudio is a multi track video editor that works entirely inside the browser without any back end services supporting or doing any kind of heavy lifting. One of the most resource intensive features of an editor is in fact the timeline, because users need to be able to load assets and scrub back and forth along it. This article gives an overview of how the timeline works and how the RAM consumption is kept under control.
Video editing is notoriously RAM intensive, and part of that difficulty is a question with no comfortable answer: how do you let a user preview a file that is larger than the total available RAM, and still let them ask for gigabyte 1 or gigabyte 100 of it, without crashing the system? Under these circumstances the only way to not crash the system is to not load the entire file into RAM but stream it instead. That solves the memory problem and immediately creates a different one. For a requested time instant, instead of just drawing the frame we now need to run the whole pipeline that yields the frame to be drawn. Everything below is me buying that latency back.
The pipeline which yields a frame starts by reading a chunk of the file at the requested position, which is encoded, so we need to decode it and then display it in a two dimensional canvas in the preview. The next time the user moves the ruler ahead or back in time we need to start the process all over again. This naive implementation works, but it is not going to be efficient.
The first source of inefficiency we can tackle is the I/O round trip to disk to fetch encoded chunks, in the case where the user happened to scrub nearby rather than really far ahead. The way to do this is to have a sample cache which responds to requests for chunks out of RAM. Caching encoded samples works fairly well here because the encoded samples are not full frames and therefore cost a fraction of the RAM to hold. The arithmetic is worth doing once: a 1080p frame sitting in a canvas is 1920 by 1080 by 4 bytes, so roughly 8MB, while the encoded packet it came out of is more likely to be a few tens of kilobytes. Two orders of magnitude for the same moment in the video, which is why you can afford to keep a lot of one and very little of the other.
The next optimization to go after is keeping some decoded frames in RAM as well, in the proximity of the sought position, in case the user seeks nearby, since decoding is not free either. Here the 8MB figure sets the budget, so the window stays deliberately narrow. If they seek beyond your cached bounds then you have no choice but to start the pipeline again.
Restarting the pipeline is not cheap, because once more you have to reach to disk to fetch the chunks, decode, possibly apply more effects and then draw, so we should only do it when it is really necessary. And it is in fact likely that users will seek to truly random positions along the timeline, in short succession of one another. Here we can do something smart: the last seek wins. Vidstudio does this by debouncing the seek operation for a few ms and tracking the id of the most recently started decode operation, then cancelling anything that started before it. That way only the latest seek operation ends up consuming resources on the end user's device.
For a client side application, being frugal with the end user's resources matters, but so does the experience, and the two are in direct competition. Every one of the decisions above is the same trade made in a different place. Cache the cheap thing generously, cache the expensive thing narrowly, and throw away work the user has already changed their mind about. What the timeline does not do yet is anything clever about where in the file it starts decoding from, which turns out to be the largest cost of all. That one needs its own article.
If you want to check out the project

Online Video Editor (No Upload, No Signup) | VidStudio

Free online video editor that runs in your browser. Multi-track timeline, frame-accurate seek, powered by WebCodecs and FFmpeg WASM. Your files never leave your device.

vidstudio.app

Free Text-Based Video Editor (No Signup, No Upload) | VidStudio

Edit video by editing the transcript, free and in your browser. Delete a word and the video cuts to match. Transcription runs on your device, nothing uploads, and there is no signup or watermark.

vidstudio.app