FFmpeg needs no introduction: it is the engine room of everything multimedia, from every player to every transcoding workflow, and VLC uses libavcodec and libavformat heavily. The FFmpeg and VideoLAN communities have always been close and share quite a few developers, often meeting at FOSDEM and VDD.
FFmpeg 9.0 is a major release. It comes four and a half months after 8.1 “Hoare”, with more than 2200 commits from over 160 authors, touching 1781 files, adding almost 85000 lines and deleting more than 33000. It bumps the major version of all seven libraries (libavutil 61, libavcodec 63, libavformat 63, libavdevice 63, libavfilter 12, libswscale 10, libswresample 7), which means an ABI break across the board, and it means the big cleanup that FFmpeg only allows itself at major bumps.
Let me go through it, subsystem by subsystem.
The most important long-term work in this release is invisible on a feature list: the multi-year rewrite of swscale, led by Niklas Haas, who landed more than 300 commits in libswscale alone this cycle, with Ramiro Polla close behind on the ARM side.
The old swscale is twenty years of bespoke, format-specific conversion code. The new architecture is completely different: every conversion is decomposed into a list of elementary operations (read, swizzle, linear transform, scale, pack…), an optimizer simplifies and splits that list, and a backend then compiles it into a chain of kernels.
In 9.0 this becomes user-visible: the public API gains an SwsBackend selector, and the backends now include the template-based C reference, a fast memcpy path, chained x86 SIMD kernels, chained AArch64 NEON kernels, and a Vulkan SPIR-V backend that compiles the same operation lists into compute shaders, so the exact same conversion graph can run on the CPU or on the GPU. There is also a new SwsScaler enum to pick the scaling algorithm explicitly, instead of the old flags soup.
The correctness engineering underneath is the part I like. The constant math driving every conversion moved to a new 64-bit rational type, computed exactly instead of in floating point, which let a whole series of overflow checks simply disappear. Ask for SWS_BITEXACT and even the Vulkan backend complies, down to decorating its linear arithmetic with SPIR-V NoContraction so the GPU is not allowed to fuse multiply-adds; leave it off and you get a faster matrix-multiply variant instead. And the optimizer learned to split operation lists per plane and into linked subpasses, so a planar conversion compiles into independent kernels per plane instead of one monolithic loop.
The new paths are still gated behind SWS_UNSTABLE, with the legacy code as the stable default. This cycle also brought palette (PAL8) support to the new architecture, with an AVX2 path for palette reads, and proper modelling of interlaced content in the format negotiation. The architecture is now in place.
You should look at Niklas’ talk at the last VDD about this topic.
More than 700 commits landed in libavcodec. The most important ones:
FFmpeg still writes assembly, and this cycle was busy: nearly 200 commits touched the x86 directories, and about a hundred the ARM ones.
On x86, a large part of the work is Andreas Rheinhardt dragging the oldest DSP code in the tree into the present: the half-pel motion compensation code (hpeldsp and fpel, which date back to the MMX era) was ported to SSE2, the H.264 intra prediction gained AVX2 horizontal predictors, the pp7 postprocessing DCT left MMX behind, and the last MMX remnants were removed from mpegvideoenc. The motion-estimation compare functions gained SSSE3 versions of the median SAD, and the new swscale x86 backend generates its SIMD kernels from NASM macros, including the AVX2 palette-read path mentioned above.
On ARM, the NEON yuv2rgb paths were reworked to process two lines at a time, across packed RGB, planar GBR, 16-bit RGB and yuva420p outputs, including big-endian 16-bit formats. HEVC intra angular modes 10 and 26 got NEON implementations, and the AAC SBR and float DSP loops were re-unrolled to better fill modern pipelines. But the most interesting ARM work hides in swscale. Ramiro Polla built the new AArch64 backend on top of rasm, a small assembler framework with an instruction-level IR and builder API. It generates the chained NEON kernels as assembler text at build time, and is designed to allow runtime code generation later.
RISC-V was quieter in FFmpeg this cycle, with RVV optimizations for hevc_add_res and the pixelutils SAD, plus portability fixes. Most of the RISC-V vector energy in the multimedia ecosystem is currently going into dav1d, where FFmpeg gets it back for free through its dav1d-based AV1 decoding.
One of the themes of this release is camera raw video in open source.
The native ProRes RAW decoder, introduced in 8.0, was synchronized with the reference implementation this cycle: it now parses the linearization curve from the bitstream, sets frame cropping, and exports the raw Bayer color parameters through a new AV_FRAME_DATA_RAW_COLOR_PARAMS side data, designed to be codec-specific because ARRIRAW is explicitly on the roadmap. A GPU-accelerated Vulkan decoder is available, as is new VideoToolbox decoding on Apple platforms.
On the other side of the pipeline, FFV1 learned to encode Bayer pixel formats: raw sensor mosaics, compressed losslessly, without debayering first. Lynne implemented it in the native codec, with a per-slice search for the best reversible colour transform coefficients, and in a separate GPU-accelerated Vulkan FFV1 encoder and decoder (yes, FFmpeg encodes and decodes FFV1 on the GPU now). The search runs as its own compute pass. Part of the work was sponsored by the Sovereign Tech Fund. The Vulkan encoder also gained a 32-bit float RGB path with a Rice-plus-remap coding mode, and takes care to preserve denormals when reading raw 16-bit floats. FFV1 is widely used for preservation masters, and losslessly compressing camera sensor data at GPU speed is a useful new capability for digital preservation.
The Vulkan work keeps expanding across the codec and filter layers:
Vulkan is not the only GPU API in town. AMD’s AMF gains a frame rate converter filter (frc_amf), HDR capabilities in its colour converter (vpp_amf, with new public helpers to translate HDR metadata to and from the AMF representation), and hardware memory mapping support. And the CUDA pipeline gains a new transpose_cuda filter, a generic filtering path in scale_cuda, and end-to-end handling of the 10- and 12-bit 4:2:2 and 4:4:4 formats that recent NVIDIA hardware can finally decode. These professional formats are mapped through NVDEC, the CUDA filters and NVENC alike.
Ten days ago, I wrote about libbluray 1.5.0 exposing Dolby Vision tracks on UHD Blu-rays. FFmpeg 9.0 picks up the other end of that pipeline, and this time the story is complete.
Dolby Vision Profile 7, the flavour used on UHD Blu-ray discs, is a strange beast: a full enhancement-layer HEVC bitstream and the RPU metadata are interleaved inside the base-layer stream, hidden in unspecified NAL unit types 62 and 63. The new dovi_split bitstream filter takes such a stream apart: keep the base layer only and get a plain, standard HEVC stream, or properly separate the layers. Around it, libavformat gained a real model for the thing: a Dolby Vision stream group (a generalization of the LCEVC stream groups into AVStreamGroupLayeredVideo), detected when demuxing MP4, MPEG-TS and Matroska, and written out again, including the hvcE enhancement-layer configuration, when muxing to MP4 or Matroska. This makes it possible to demux, split and remux these streams while keeping the signaling intact.
On the metadata front, 9.0 adds parsing, writing and passthrough for SMPTE 2094-50 dynamic HDR metadata, another SMPTE 2094 format alongside HDR10+, down to encoding with libaom and libvpx, Matroska block additions, and ffprobe output. The spec is being finalized in the open, on GitHub of all places. Yes, SMPTE drafts standards on GitHub now. The whole ITU-T T.35 dynamic metadata zoo, including HDR10+, China’s HDR Vivid, AOM film grain and Active Format Description, is now parsed through one unified set of helpers instead of being reimplemented codec by codec. And if your file carries wrong static HDR metadata, the ffmpeg CLI gains per-stream -mastering_display and -content_light input overrides to correct the metadata on input, which anyone who has handled badly-mastered HDR files will appreciate.
Close to 400 commits landed in libavformat. Beyond the Dolby Vision work above:
Everything deprecated during the 8.x cycle is gone, and the list is long.
On the command line, -vsync is finally removed (use -fps_mode), along with -top, -qphist, -filter_complex_script and -adrift_threshold. The OpenMAX encoders are removed. The NPP filters (scale_npp, scale2ref_npp, sharpen_npp, transpose_npp) are gone with the rest of the libnpp support, deprecated last September; the transpose_cuda filter mentioned above picks up where transpose_npp left off, without the external dependency. The packed-YUV oddities v308/v408/v410 lose their dedicated codecs. And NVENC drops the old preset aliases, the legacy rate-control modes and support for SDKs older than 11.1.
One behaviour change deserves everyone’s attention: FFmpeg now verifies TLS peer certificates by default. This was announced with 8.0 and applied at this bump, exactly as promised. If your scripts talk to servers with broken certificates, they were living on borrowed time anyway.
My two favourite removals, though, are pure FFmpeg lore:
The -re/-readrate machinery was reworked to throttle on the slowest stream instead of the fastest, skipping finished and not-yet-started streams. If you have ever seen a live restream starve because of a sparse subtitle track, this is your fix. ffprobe learned to print the new side data types (SMPTE 2094-50, IAMF mix gain and demixing parameters, downmix info), and the ffmpeg CLI can now create LCEVC stream groups when muxing.
And the DNN filters gain an ONNX Runtime backend with CUDA and DirectML execution providers, so you can run ONNX models directly in a filter graph for super-resolution, denoising or background removal, without converting them for TensorFlow or OpenVINO first. Combined with the Whisper filter from 8.0, the filter graph is becoming a decent place to run inference next to your decode loop.
Some less visible changes are worth mentioning too:
FFmpeg 9.0 is a substantial release, with new features as well as a lot of cleanup. The ABI break means distributions and applications will take some time to migrate the seven new sonames, and VLC will follow.
The full Changelog is worth a read, and so is the git log; congratulations to the 160+ people who wrote this release, and in particular to Niklas Haas, Andreas Rheinhardt, Michael Niedermayer, James Almer, Lynne, Marvin Scholz and Ramiro Polla, who top a very long list, several of them doing part of this work at FFlabs and/or VideoLabs.
The code is, as always, open source. Patches welcome.
Source: Hacker News — This article was automatically imported from the source. Read full article at original source →
