From 98c700390d6a6d43b1f1531afc908106630c6de4 Mon Sep 17 00:00:00 2001 From: mikael-lovqvists-claude-agent Date: Sat, 28 Mar 2026 20:37:06 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20clarify=20MJPEG=20decode=20path=20?= =?UTF-8?q?=E2=80=94=20tjDecompressToYUV2=20+=20YUV=20textures?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit libjpeg-turbo can decompress directly to planar YUV, bypassing CPU-side color conversion entirely. Document the precise pipeline: separate Y/Cb/Cr GL_RED textures, BT.601 matrix in fragment shader, SIMD Huffman+DCT on CPU only. Co-Authored-By: Claude Sonnet 4.6 --- architecture.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/architecture.md b/architecture.md index 7d9ed9e..1def3ea 100644 --- a/architecture.md +++ b/architecture.md @@ -407,12 +407,15 @@ The initial implementation uses **GLFW** for window and input management and **O GLFW handles window creation, the event loop, resize, and input callbacks — it also supports Vulkan surface creation using the same API, which makes a future renderer swap straightforward. Input events (keyboard, mouse) are normalised by GLFW before being encoded as protocol messages. The OpenGL renderer: -1. Receives a decoded frame as a pixel buffer (libjpeg-turbo for MJPEG, raw for uncompressed formats) -2. Uploads it as a 2D texture -3. Runs a fragment shader that handles YUV→RGB conversion (where needed) and scaling/filtering -4. Presents via GLFW's swap-buffers call +1. For **MJPEG**: calls `tjDecompressToYUV2` (libjpeg-turbo) to decompress directly to planar YUV — no CPU-side color conversion. JPEG stores YCbCr internally so this is the minimal decode path: Huffman + DCT output lands directly in YUV planes. +2. Uploads Y, Cb, Cr as separate `GL_RED` textures (chroma at half resolution for 4:2:0 / 4:2:2 as delivered by most V4L2 cameras). +3. Fragment shader samples the three planes and applies the BT.601 matrix to produce RGB — a few lines of GLSL. +4. Scaling and filtering happen in the same shader pass. +5. Presents via GLFW's swap-buffers call. -This keeps CPU load low — chroma conversion and scaling happen on the GPU — while keeping the implementation simple relative to a full Vulkan pipeline. +For **raw pixel formats** (BGRA, YUV planar from the wire): uploaded directly without decode; shader handles any necessary swizzle or conversion. + +This keeps CPU load minimal — the only CPU work for MJPEG is Huffman decode and DCT, which libjpeg-turbo runs with SIMD. All color conversion and scaling is on the GPU. #### Renderer: Vulkan (future alternative)