mikael-lovqvists-claude-agent/video-setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make V4L2 dequeue the primary ingest path; demote EOI scanner

Browse Source 
ingest module: dequeue V4L2 buffers, emit one encapsulated frame per buffer.
Driver guarantees per-buffer framing for V4L2_PIX_FMT_MJPEG; no scanning needed.

mjpeg_scan: future optional module for non-compliant hardware only.
Explicitly a workaround, not part of the primary pipeline.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
mikael-lovqvists-claude-agent
2026-03-25 22:55:44 +00:00
parent caad1565b8
commit 51e2a3e79e
2 changed files with 9 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
architecture.md
View File

@@ -84,30 +84,21 @@ graph LR
CTRL -.->|node control| RELAY
```
The Pi runs only:
- A forwarding process (e.g. `dd if=/dev/video0 | nc <host> <port>` or equivalent)
- The V4L2 control endpoint (receives parameter change commands)
The Pi runs a node process that dequeues V4L2 buffers and forwards each buffer as an encapsulated frame over TCP. It also exposes the V4L2 control endpoint for remote parameter adjustment.
Everything else happens on machines with adequate resources.
### Frame Boundaries and the Ingest Scanner
### V4L2 Buffer Dequeuing
When the V4L2 driver is well-behaved and delivers `V4L2_PIX_FMT_MJPEG`, each dequeued buffer contains exactly one complete MJPEG frame — the driver guarantees this boundary. If the Pi runs a proper node process that dequeues buffers and sends each one as an encapsulated frame, boundaries are preserved across the wire and no scanning is needed at the ingest end.
When a V4L2 device is configured for `V4L2_PIX_FMT_MJPEG`, the driver delivers one complete MJPEG frame per dequeued buffer — frame boundaries are guaranteed at the source. The ingest module dequeues these buffers and emits each one as an encapsulated frame directly into the transport. No scanning or frame boundary detection is needed.
When the Pi uses a raw pipe (`dd | nc`), the buffer boundaries are discarded at the source. What arrives at the ingest node is a concatenated MJPEG byte stream with no framing. The ingest module's two-pass EOI state machine exists specifically for this case: it scans the incoming bytes for JPEG SOI (`0xFF 0xD8`) and EOI (`0xFF 0xD9`) markers to recover frame boundaries from the raw stream and re-emit discrete frames into the encapsulated transport.
This is the primary capture path. It is clean, well-defined, and relies on standard V4L2 kernel behaviour rather than heuristics.
This is the **primary and planned** ingest path for the Pi forwarding scenario. It is not a hack — it is the correct tool when the source is an opaque byte stream carrying concatenated MJPEG.
### Misbehaving Hardware: `mjpeg_scan` (Future)
### Fallback: Non-Standard and Containerised Sources
Some hardware does not honour the per-buffer framing contract — cheap USB webcams or cameras with unusual firmware may concatenate multiple partial frames into a single buffer, or split one frame across multiple buffers. For these cases a separate optional `mjpeg_scan` module provides a fallback: it scans the incoming byte stream for JPEG SOI (`0xFF 0xD8`) and EOI (`0xFF 0xD9`) markers to recover frame boundaries heuristically.
Some sources produce MJPEG in unexpected ways:
- Cheap USB webcams with firmware that wraps frames in an AVI or RIFF container even on the V4L2 interface
- IP/network cameras streaming via HTTP multipart MIME (`multipart/x-mixed-replace`) or RTSP with unusual packetisation
- Any source where the JPEG payload is embedded inside another container format
For these cases the preferred fallback is to route through **ffmpeg**, which handles essentially any container format and emits clean decoded frames or re-muxed output. A custom parser for every quirky format is not a worthwhile investment.
The EOI scanner in the ingest module is kept as a secondary option for sources that are known to produce a raw concatenated MJPEG byte stream but where spawning ffmpeg is undesirable (resource constraints, startup latency, etc.). It is not the general solution for malformed or unusual sources.
This module is explicitly a workaround for non-compliant hardware. It is not part of the primary pipeline and will be implemented only if a specific device requires it. For sources with unusual container formats (AVI-wrapped MJPEG, HTTP multipart, RTSP with quirky packetisation), the preferred approach is to route through ffmpeg rather than write a custom parser.
---