PixelSmash (CVE-2026-8461): How a Rounding Mismatch in FFmpeg's MagicYUV Decoder Pops Jellyfin Through an Auto-Library Scan

Once a quarter, something lands that reminds us how thin the line is between "we transcoded a video" and "we ran a shell on the host that did it." This week's reminder comes from JFrog Security Research and a heap bug in FFmpeg's MagicYUV decoder. The exploit is roughly 50 KB; the trigger is whatever automatic library scan or thumbnail job your media server already runs on uploads. If your Jellyfin or Nextcloud is unpatched, an attacker who can drop a video file into a watched folder can probably get a shell on the host that holds the rest of your media, your share tokens, and the OAuth bearer your reverse proxy hands every other service on the box.

What pulled us in was not the impact, which is the usual "open this file, lose this host." It was the root cause. Two functions inside FFmpeg need to agree on a single number, the height of a frame's chroma plane, and they compute it slightly differently. That small disagreement is enough to walk past the end of a heap allocation and land on a function pointer.

The Rounding Mismatch at the Root

FFmpeg's frame allocator and its MagicYUV decoder both need the height of the chroma plane. For chroma-subsampled formats like YUV420P, that height is half the luma height. When the luma height is odd, half of odd is not an integer, so each side has to choose a rounding rule, and the two sides choose differently. The allocator rounds once per frame; the decoder rounds per slice and then accumulates the difference across every slice that follows. Everything else in the exploit is plumbing built on top of that single inconsistency.

A MagicYUV-encoded video can be split into slices, and each slice carries its own slice_height value in the bitstream. The decoder applies AV_CEIL_RSHIFT(slice_height, vsub) per slice, where vsub is the chroma vertical subsample factor. Two slices of height 31 and 1 inside a 32-tall frame should chroma-decode into 16 rows. With ceiling rounding applied per slice, AV_CEIL_RSHIFT(31, 1) is 16 and AV_CEIL_RSHIFT(1, 1) is 1, giving 17 chroma rows written into a 16-row buffer, one extra row per slice. Multiplied by the chroma stride, that reaches JFrog's figure of roughly 640 attacker-controlled bytes written past the end of the chroma plane.

From Overflow to Function Pointer

Heap allocators pack objects next to each other, and FFmpeg's AVBuffer struct, the refcounted handle it uses to manage every buffer's lifetime, sits in memory immediately after the chroma plane's pixel data. That struct holds a function pointer named buf->free, the destructor called when the refcount hits zero. The overflow scribbles directly over it, and the value the attacker writes is the address of system().

The trigger is automatic. Jellyfin scans your library for new files, the scan calls ffprobe to read metadata, ffprobe calls the MagicYUV decoder, and the decoder runs the out-of-bounds write. Cleanup then invokes av_buffer_unref, which dispatches through buf->free. That dispatched function is now system(), called with an attacker-shaped argument sitting in the slot the overflow filled. The child shell runs its command before the parent process eventually segfaults, by which point the media host has already executed whatever curl-pipe-to-bash the attacker wanted. JFrog confirmed the chain end to end against Jellyfin 10.11.9 and Nextcloud's video preview generator. ASLR is the only thing standing between "the box crashed" and "the box is theirs," and we have already triaged installs where it had been quietly turned down for a benchmarking experiment and never put back. This is a confused-deputy of memory safety, where one component trusts a dimension another component derived under a different rule.

The Shared libavcodec Blast Radius

The same flaw reaches far past media servers because almost everything that touches video links the same libavcodec. Jellyfin, Emby, Nextcloud, Immich, PhotoPrism, OBS Studio, Kodi, mpv, and any GNOME or KDE file manager using ffmpegthumbnailer all share that library, and JFrog confirmed crashes on every one of them. Any code path that derives a length or a dimension from untrusted input in one place and recomputes it elsewhere is a candidate for the same class of overflow, which is why the patched package alone does not settle the question of whether your fleet is actually safe.

Why This Slips Past Normal Tooling

This is where most blue teams are blind, because nothing about the bug looks like an attack until it is over. There is no exploit kit signature, no payload string in a URI, no /etc/passwd hiding inside a parameter. A guest uploads a perfectly valid MKV, the server transcodes it as it has transcoded a million MKVs, and the only weird thing on the host is that the FFmpeg process briefly spawned a shell. That single anomaly is the entire detection.

Hunting the Process Tree, Not the Payload

The hunt is process tree, not content. Anywhere ffmpeg or ffprobe is the parent and a shell (sh, bash, /bin/sh, dash) or a scripting interpreter is the child, you have your alert. That maps cleanly to MITRE ATT&CK T1059 Command and Scripting Interpreter, T1203 Exploitation for Client Execution when the trigger fires on a workstation, T1190 Exploit Public-Facing Application whenever the media server faces the internet, and T1574 Hijack Execution Flow for the buf->free rewrite. In Sysmon EventID 1 or any decent EDR, the rule is one line. We have not seen a legitimate workflow where ffprobe execs bash, and if yours has one, this is a fine week to write down which one.

Two more places to look. MagicYUV uses FourCC codes like M8RG and M8Y0 in AVI containers and equivalent codec tags in MKV and MOV; if you do not transcode MagicYUV in your fleet, blocking it at the upload gateway removes the attack surface entirely. Then confirm /proc/sys/kernel/randomize_va_space is set to 2 everywhere FFmpeg runs, since PixelSmash without ASLR is reliable RCE and with ASLR is at least a fault rather than a breach. A patched binary in your package manager is also not the same as the FFmpeg loaded inside a Jellyfin container or a snap-packaged Immich, so check the process address space, not apt.

Patch, Then Prove the Patch Held

The patched FFmpeg is version 8.1.2, released June 17, 2026. Update the host package, then rebuild or pull fresh images for every app that bundles its own FFmpeg (which is most of them). Until you are sure the running binary is patched, turn off video preview generation in Nextcloud and Immich, since the preview pipeline is the gun and the upload directory is the trigger.

Then write the EDR rule. Alert on any direct child of ffmpeg or ffprobe whose image is a shell or a scripting interpreter, and pair it with a SIEM rule for unusual outbound connections from media-server hosts following a library-scan event. Run a known-bad MagicYUV file against your patched stack and confirm the OOB write no longer fires; patches that look applied and patches that actually held are not the same thing.

This is the loop BlueTeamAutomation automates end to end: ingest the advisory, run the simulated exploit against your EDR, SIEM, and SOAR, and write the compliance evidence back so the next time a 50 KB MKV reaches your storage you find out from your own detection rather than from someone else's blog. The work below the patch advisory is the work that decides whether your next breach starts with a media file or not.