SaaS Metrics
GitHub Issue
Hardware acceleration: GPU/media-engine decode, GPU effects and GPU export
## Problem
A user on Linux with an AMD GPU reports that preview and rendering barely touch the GPU while CPU usage spikes. The same is true on macOS and Windows. FilmCraft does use the GPU today, but only for the last step of preview:
| Stage | Where it runs today |
|---|---|
| Video decode (H.264, HEVC, VP9, AV1, ProRes, DNx) | **CPU**, pure-Rust software decoders |
| Preview compositing (scale, position, opacity, YUV→RGB, dissolves) | GPU (`filmcraft-gpu`, wgpu/WGSL) |
| Effects, non-Normal blend modes, adjustment layers, nested sequences, most transitions | **CPU**: rendered per layer, then handed to the GPU as an image |
| Export rendering | **CPU** (`render::render_sequence`); the GPU compositor isn't used |
| Export encoding (H.264, ProRes, …) | **CPU**, pure-Rust encoders |
Measured (docs/performance.md): playing 4K HEVC costs about **90 CPU-ms per displayed frame, and the decoder alone accounts for 82–87 ms**. That's 2–3 cores fully busy at 24 fps while the GPU only draws the final quad. Effects and export add more CPU on top.
## Goal
On machines that support it, preview and export should use the GPU and the OS's hardware media engines (VA-API / Vulkan Video, VideoToolbox, Media Foundation / D3D11). The pure-Rust paths stay as the reference and the fallback, so FilmCraft keeps working, with identical output, everywhere else.
Principles:
- **Auto by default, always fallible.** Hardware is tried first where supported. Any error (unsupported profile, driver failure...
View Raw Thread
Developer & User Discourse
echelon • Oct 5, 2026
**Hardware decoding on macOS is in** (#33, merged).
H.264 and HEVC (8/10-bit, 4:2:0 and 4:2:2) now decode on VideoToolbox's hardware decoder. It's on by default, and Settings ▸ Playback ▸ Hardware decoding switches it off. Output is bit-exact with our software decoders on every test stream. If the hardware fails mid-stream, decoding continues transparently in software.
Apple M4 Pro, load average 150–190, Off → Auto:
| | CPU ms per frame | fps |
|---|---|---|
| H.264 2160p decode | 119 → 4.2 | 35 → 107 |
| HEVC 2160p decode | 86 → 3.7 | 49 → 217 |
| 4K H.264 playback at Full resolution (frames shown/dropped) | ~120 → ~9 | 104/88 → 192/0 |
This also adds `crates/platform`, the one crate allowed `unsafe` for OS media APIs. Linux (VA-API) and Windows (Media Foundation) backends will plug into it.
**Next:**
- effects on the GPU (in progress);
- VA-API for Linux and AMD, which needs a Linux machine to test;
- Media Foundation for Windows;
- passing decoded frames straight to the GPU wit...
H.264 and HEVC (8/10-bit, 4:2:0 and 4:2:2) now decode on VideoToolbox's hardware decoder. It's on by default, and Settings ▸ Playback ▸ Hardware decoding switches it off. Output is bit-exact with our software decoders on every test stream. If the hardware fails mid-stream, decoding continues transparently in software.
Apple M4 Pro, load average 150–190, Off → Auto:
| | CPU ms per frame | fps |
|---|---|---|
| H.264 2160p decode | 119 → 4.2 | 35 → 107 |
| HEVC 2160p decode | 86 → 3.7 | 49 → 217 |
| 4K H.264 playback at Full resolution (frames shown/dropped) | ~120 → ~9 | 104/88 → 192/0 |
This also adds `crates/platform`, the one crate allowed `unsafe` for OS media APIs. Linux (VA-API) and Windows (Media Foundation) backends will plug into it.
**Next:**
- effects on the GPU (in progress);
- VA-API for Linux and AMD, which needs a Linux machine to test;
- Media Foundation for Windows;
- passing decoded frames straight to the GPU wit...
echelon • Oct 5, 2026
**31 common effects now run on the GPU** (#35, merged).
On the GPU:
- **colour:** Brightness & Contrast, ProcAmp, Tint, Black & White, Color Balance, Levels, Gamma, Channel Mix, ASC CDL, Leave/Change/Pass/Replace Color, Invert, Posterize, Extract, Alpha Adjust;
- **geometry:** Crop, Flip, Transform, Mirror, Offset, Edge Feather;
- **blur:** Gaussian, Camera, Directional;
- **sharpen:** Sharpen, Unsharp Mask.
A clip moves to the GPU when all of its effects are on this list; otherwise it renders on the CPU as before. GPU output matches the CPU reference to within 7e-5 relative on 78 cases, including extreme parameters. On a three-layer test with effects, frame-worker time drops from 220–307 ms to 1.8–93 ms per frame.
**Done so far:** blend modes (#32), macOS hardware decode (#33), common effects (#35).
**Next:**
- VA-API (Linux) and Media Foundation (Windows) decode;
- zero-copy upload of decoded frames;
- Lumetri, keys and masks on the GPU;
- moving effect passes off the UI thread;
...
On the GPU:
- **colour:** Brightness & Contrast, ProcAmp, Tint, Black & White, Color Balance, Levels, Gamma, Channel Mix, ASC CDL, Leave/Change/Pass/Replace Color, Invert, Posterize, Extract, Alpha Adjust;
- **geometry:** Crop, Flip, Transform, Mirror, Offset, Edge Feather;
- **blur:** Gaussian, Camera, Directional;
- **sharpen:** Sharpen, Unsharp Mask.
A clip moves to the GPU when all of its effects are on this list; otherwise it renders on the CPU as before. GPU output matches the CPU reference to within 7e-5 relative on 78 cases, including extreme parameters. On a three-layer test with effects, frame-worker time drops from 220–307 ms to 1.8–93 ms per frame.
**Done so far:** blend modes (#32), macOS hardware decode (#33), common effects (#35).
**Next:**
- VA-API (Linux) and Media Foundation (Windows) decode;
- zero-copy upload of decoded frames;
- Lumetri, keys and masks on the GPU;
- moving effect passes off the UI thread;
...
echelon • Oct 6, 2026
**Correction to my earlier update on #33.** VideoToolbox hardware decoding was **not** active in the desktop app until #41 (merged today, 5c8471c).
The desktop app called `filmcraft_platform::register()` only as an argument to `log::info!`. No logger is installed, so the `log` crate never evaluated it and the hardware decoder was never registered. The CLI/MCP and the benchmark harness call `register()` directly. That's why the parity tests and the benchmark numbers in #33 were real, and also why we didn't notice that desktop users, v0.2.0 and v0.2.1 included, still decoded in software.
#41 (by @justjohn12345) registers it unconditionally at startup and adds a test that it happens without a logger. The next release is the first where desktop users get hardware decoding by default. I missed this in review; thanks to the contributor for catching it.
The desktop app called `filmcraft_platform::register()` only as an argument to `log::info!`. No logger is installed, so the `log` crate never evaluated it and the hardware decoder was never registered. The CLI/MCP and the benchmark harness call `register()` directly. That's why the parity tests and the benchmark numbers in #33 were real, and also why we didn't notice that desktop users, v0.2.0 and v0.2.1 included, still decoded in software.
#41 (by @justjohn12345) registers it unconditionally at startup and adds a test that it happens without a logger. The next release is the first where desktop users get hardware decoding by default. I missed this in review; thanks to the contributor for catching it.
All 27 blend modes composite in WGSL, Dissolve included. A clip with a blend mode no longer forces the whole frame through the CPU renderer. The GPU output matches the CPU reference: every mode is within one half-float step on the edge-case tests, and 99% of interior pixels are within one 8-bit level on full sequences.
On a 1080p three-layer stack with Screen and Overlay, playing at Full resolution:
| | shown/dropped frames | CPU ms per frame |
|---|---|---|
| before | 19/173 and 138/54 | 168–214 |
| after | 191/1 and 192/0 | 86–87 |
That's the same cost as the plain Normal stack.
**Coming next:**
- **#33** adds hardware decoding on macOS through VideoToolbox, plus the `crates/platform` FFI crate that the Linux (VA-API) and Windows (Media Foundation) backends will use. It's in final CI. 4K H.264 decode drops from 119 to 4 CPU-ms per frame.
- The most common effects (colour adjustments, crop, flip, transform, Gaussian blur...