The analysis tape: our posterized video preview

Why the tape exists

A marly.studio job starts with a long upload - a two-hour VOD, a four-hour recording - and ends with a list of reviewed clip candidates, each carrying the reason it was picked. Between those two points the AI finder is reading a transcript chunk by chunk and the reviewer model is grading what it finds. That takes minutes, and a spinner for those minutes would tell you nothing: not whether the upload decoded, not whether the audio transcribed, not whether the worker is on chunk 2 or chunk 40.

So the job page plays a tape instead: a live, monochrome, moving picture of your actual footage, published piece by piece while the analysis runs. It answers the three questions a spinner cannot - did my file upload, is it the right video, and is it still working - and it does it in a look that cannot be confused with the finished clips.

What you are looking at

The tape is a canvas, not a video element. Every frame you see has already been shrunk, quantized, and re-composited by hand - the footage underneath is only raw material. Two frame sources feed a real job - the filmstrip first, then the moving proxy as it lands - plus a third that exists only on demo tapes:

1. 01The filmstrip. A single JPEG grid of stills from the source, drawn first and used to crossfade between keyframes while the smooth proxy catches up.
2. 02The motion proxy. A low-resolution h264 video the worker transcodes alongside the analysis - this is what makes the tape move. Sources longer than four minutes are sliced into 120-second publishes so the early minutes arrive first; shorter jobs are a single pass.
3. 03Generated dot motion (demo tapes only). On showcase jobs the player draws a synthetic scene - grid, drift bars, a tracking reticle, a face box, a scan line - into the same canvas. A real job never invents frames: before footage lands it says it is still waiting, and if capture fails it says unavailable.

All three end at the same place: a tiny canvas, long edge capped at 160px, run through the four-tone pass described below.

What the worker cuts

Two ffmpeg passes run beside the real analysis. Neither is allowed to slow the job down, so both are engineered for speed over fidelity - fidelity is pointless anyway, because the client is about to throw most of the pixels away.

The filmstrip

One frame every intervalSeconds - one second on dense jobs (≤10 min can exceed the usual 240-tile budget), two or more on longer ones - scaled and letterboxed into a 320px box (landscape defaults to 320×180; portrait flips) and packed 12 columns wide into a single JPEG:

fps=1/${intervalSeconds},
scale=320:180:force_original_aspect_ratio=decrease,
pad=320:180:(ow-iw)/2:(oh-ih)/2:black,
tile=12xN

The decode flag that makes this cheap is -skip_frame nokey: ffmpeg reads keyframes only, so grabbing 240 stills from a multi-hour file takes seconds instead of a full decode. Nearest-keyframe accuracy is fine for a review strip.

The motion proxy

A second ffmpeg transcodes the source to a ~320px MP4 at the source frame rate (clamped to 30 - the tape is a preview, not a deliverable), encoded libx264 -preset ultrafast. Only sources longer than two segment spans (240 seconds) are cut into 120-second segments published as each finishes; shorter jobs get a single encode. The comment in the recorder says it plainly: encode quality is invisible after the client's 160px posterize.

Alongside the media, the worker publishes a JSON artifact: the chunk timeline, every candidate moment, transcript lines, and a downsampled RMS loudness track. The player drives its playhead, its rolling transcript window, and a live audio VU bar off that one document.

The four-tone pass

This is the part people ask about. Every rendered frame - filmstrip tile, motion frame, or generated scene - finishes with an in-place pixel rewrite:

function posterize2Bit(data: Uint8ClampedArray, w: number, h: number) {
  const n = w * h;
  for (let p = 0, o = 0; p < n; p += 1, o += 4) {
    const sourceLum = 0.299 * data[o] + 0.587 * data[o + 1] + 0.114 * data[o + 2];
    const lum = GAMMA_LUT[sourceLum | 0];
    const tone = Math.round(lum / POSTERIZE_STEP) * POSTERIZE_STEP;
    data[o] = tone; data[o + 1] = tone; data[o + 2] = tone;
    data[o + 3] = 255;
  }
}

Three deliberate choices live inside those ten lines:

1. 01Rec.601 luma. 0.299 R + 0.587 G + 0.114 B - the standard video-weighted luminance, so what reads bright to your eye reads bright in the render.
2. 02A gamma-0.8 lift through a lookup table. 255 * pow(v/255, 0.8) is precomputed for all 256 inputs once, so the per-pixel pass is a table read rather than a Math.pow per channel. The lift exists for the footage we actually see most: dark gaming scenes would crush to pure black without it.
3. 03Snap, don't scatter. POSTERIZE_STEP is 85, so every pixel lands on 0, 85, 170, or 255 - a flat 2-bit palette, a true posterize with no Bayer matrix and no error diffusion. We tried a scattered-dot approach through the mid-tones first; the grain read as compression damage on small screens. Clean tone bands won.

The last trick is free: the canvas is capped at 160px on its long edge and CSS scales it back up with image-rendering: pixelated. The chunky Game Boy texture costs nothing - it is just the browser refusing to interpolate.

Motion and polish

While the motion proxy is still transcoding, the tape cannot stall - so the filmstrip fills in by crossfading between adjacent tiles. A fractional tile index picks the two nearest stills and alpha-blends them (globalAlpha = tileFloat - tile), which turns a 1fps strip into readable motion instead of a slideshow.

On top of the footage sits a second canvas of chrome: a faint 8×5 grid, a live VU bar on the right edge driven by the artifact's RMS energy track, and a rolling three-line transcript so you can watch the words the AI is hearing. When the job finishes, the whole stage runs an old-television power-off - the picture collapses to a bright horizontal line, pinches to a point, and fades (tape-crt-off, 1.2s) before the results grid takes over.

Why it stays fast

Every shortcut is the same shortcut: spend bytes and CPU where the eye can still see the result, nowhere else. The motion proxy is ultrafast h264 at 320px because four tones hide encode artifacts completely. The canvas is 160px because per-pixel JS at 60fps needs a small budget - roughly 16,000 pixels a frame, not a quarter-million. The gamma curve is a 256-entry Float32Array because a lookup beats a pow() per channel per pixel. And the redraw loop gates itself on a 16ms interval rather than every animation frame event, so a 120Hz display does not double the work.

The effect works because it is cheap, not in spite of it: the same constraints that keep it light on a low-end laptop are what give it the texture.

FAQ