video-setup/planning.md

Planning

Approach

Build the system module by module in C11. Each module is a translation unit (.h + .c) with a clearly defined API. Modules are exercised by small driver programs in dev/ before anything depends on them. This keeps each unit independently testable and prevents architectural decisions from being made prematurely in code.

The final binary is a single configurable node program. That integration work comes after the modules are solid.

---

Directory Structure

video-setup/
  src/
    modules/
      common/        - shared definitions (error types, base types)
      config/        - INI file loader with schema-driven defaults, typed getters
      media_ctrl/    - Linux Media Controller API (topology, pad formats, links)
      v4l2_ctrl/     - V4L2 camera controls (enumerate, get, set)
      serial/        - little-endian binary serialization primitives
      transport/     - framed TCP stream, single-write send
      discovery/     - UDP multicast announcements, peer table, found/lost callbacks
      protocol/      - typed write_*/read_* message functions
      test_image/    - test pattern generator (colour bars, ramp, grid; YUV420/BGRA)
      xorg/          - GLFW+OpenGL viewer sink; stub for headless builds
      reconciler/    - generic wanted/current state machine reconciler
      ingest/        - V4L2 capture loop, MMAP buffers, on_frame callback
    node/            - video node binary (source + display sink roles)
  include/           - public headers
  dev/
    cli/             - exploratory CLI drivers, one per module
    web/             - development web UI (Node.js/Express); browser-side equivalent
                       of the CLI tools; depends on protocol being finalised
    experiments/     - freeform experiments
  tools/
    gen_font_atlas/  - build-time bitmap font atlas generator (Python/Pillow);
                       outputs build/gen/font_atlas.h consumed by xorg module
  tests/             - automated tests (later)
  Makefile
  architecture.md
  planning.md
  conventions.md

---

Module Order

Modules are listed in intended build order. Each depends only on modules above it.

#	Module	Status	Notes
1	common	done	Error types, base definitions — no dependencies
—	config	done	INI file loader with schema-driven defaults, typed getters, FLAGS type for bitmask values
2	media_ctrl	done	Media Controller API — device and topology enumeration, pad format config
3	v4l2_ctrl	done	V4L2 controls — enumerate, get, set camera parameters
4	serial	done	put/get primitives for little-endian binary serialization into byte buffers
5	transport	done	Encapsulated transport — frame header, TCP stream abstraction, single-write send
6	discovery	done	UDP multicast announcements, peer table, found/lost callbacks
7	protocol	done	Typed write_*/read_* functions for all message types; builds on serial + transport
8	test_image	done	Test pattern generator — colour bars, luminance ramp, grid crosshatch; YUV420/BGRA output
9	xorg	done	GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented; viewer controls (zoom, pan, scale policy) not yet exposed remotely
10	reconciler	done	Generic wanted/current state machine reconciler — resource state graphs, BFS pathfinding, event + periodic tick; used by node to manage V4L2 devices, transport connections, and future resources (codec processes etc.)
11	ingest	done	V4L2 capture loop — open device, negotiate MJPEG format, MMAP buffers, capture thread with on_frame callback; start/stop lifecycle managed by reconciler
—	node	done	Video node binary — config, discovery, transport server, V4L2/media control request handlers; display sink role (START_DISPLAY/STOP_DISPLAY handlers, multi-window xorg viewer, declarative display slot reconciler)
12	frame_alloc	not started	Per-frame allocation with bookkeeping (byte budget, ref counting)
13	relay	not started	Input dispatch to output queues (low-latency and completeness modes)
14	archive	not started	Write frames to disk, control messages to binary log
15	codec	not started	Per-frame encode/decode — MJPEG (libjpeg-turbo), QOI, ZSTD-raw, VA-API H.264 intra; used by screen grab source and archive
16	web node	not started	Node.js/Express peer — speaks binary protocol on socket side, HTTP/WebSocket to browser; protocol.mjs mirrors C protocol module
—	mjpeg_scan	future	EOI marker scanner for misbehaving hardware that does not deliver clean per-buffer frames; not part of the primary pipeline

---

Dev Tools

CLI (dev/cli/)

Each module gets a corresponding CLI driver that exercises its API and serves as both an integration check and a useful development tool.

Driver	Exercises	Notes
media_ctrl_cli	media_ctrl	List media devices, show topology, configure pad formats
v4l2_ctrl_cli	v4l2_ctrl	List controls, get/set values — lightweight v4l2-ctl equivalent
transport_cli	transport	Send/receive framed messages, inspect headers
test_image_cli	test_image	Generate test patterns, write PPM for visual inspection
xorg_cli	xorg	Display test pattern in viewer window; exercises scale/anchor modes and text overlays
v4l2_view_cli	V4L2 + xorg	Live camera viewer — auto-selects highest-FPS format, FPS/format overlay; bypasses node system
stream_send_cli	V4L2 + transport + protocol	Capture MJPEG from V4L2, connect to receiver, send VIDEO_FRAME messages; prints fps/Mbps stats
stream_recv_cli	transport + protocol + xorg	Listen for incoming VIDEO_FRAME stream, display in viewer; fps/Mbps overlay; threaded transport→GL handoff
reconciler_cli	reconciler	Simulated state machine experiment — define resources with fake transitions, drive reconciler via CLI commands; validates the generic reconciler before wiring into the node
controller_cli	transport + protocol + discovery	Interactive controller REPL — connects to nodes by peer index or host:port; supports enum-devices, enum-controls, get/set-control, start-ingest, stop-ingest, start-display, stop-display; readline + discovery integration; temporary dev tool — will be superseded by a dedicated controller binary that holds simultaneous connections to all peers

Web UI (dev/web/)

A Node.js/Express development web UI that connects to running video nodes as a binary protocol peer. Exposes V4L2 control inspection and adjustment, media topology view, and stream state through a browser interface. Supports live peer discovery via SSE — discovered nodes appear automatically in the UI.

This is a development aid, not the production dashboard. The production dashboard (full stream configuration UI) is a later, separate project.

---

Future: Protocol Preprocessor

The C protocol module and JavaScript protocol.mjs will eventually be generated from a single schema by a future preprocessor. This eliminates drift between the two implementations. The preprocessor also handles error location codes (see common/error). Neither the schema format nor the preprocessor tool exists yet — the hand-written implementations are the interim state.

---

Deferred Decisions

These are open questions tracked in architecture.md that do not need to be resolved before module work begins:

• Graph representation format
• Connection establishment model (push vs pull)
• Completeness queue drop policy (oldest vs newest, per-output config)
• Stream ID remapping across relay hops
• Transport for relay edges (TCP / UDP / shared memory)
• Node discovery mechanism
• Hard vs soft byte budget limits
• Cooperative capture release: if a capture source has no live downstream targets for a configurable time window, stop capture and release the device. Intended as a resource-conservation policy rather than an immediate reaction to disconnect events. Requires the node to track downstream liveness (e.g. last successful send timestamp per output) and implement a reaper timer.
• Unified device model: active display windows should be registered as devices alongside V4L2 cameras, using the same ENUM_DEVICES / ENUM_CONTROLS / GET_CONTROL / SET_CONTROL protocol. START_DISPLAY would return a device_id for the opened window; controls (scale, anchor, position, size, zoom, pan) are then addressable as (device_id, control_id) pairs like any other device. Requires a device_type field in ENUM_DEVICES responses so controllers can distinguish V4L2 devices from display windows. Future device types: codec processes, screen grab sources. This extends naturally to shader-based post-processing and other viewer state as controls.
• controller_cli is a temporary dev tool; the long-term replacement is a dedicated controller binary outside dev/cli/ that maintains simultaneous connections to all discovered nodes (not switching between them). Commands address a specific node by peer index. This mirrors the web UI's model of administering the whole network rather than one node at a time. The connect / active-connection model in the current controller_cli is an interim design choice that should not be carried forward.
• start-ingest peer addressing: the dest_host + dest_port in START_INGEST is awkward to type manually and requires the caller to know the target's TCP port. Should accept a peer ID (index from the discovered peer table on the node) so the node can resolve the address itself. Requires the node to run discovery and expose its peer table.
• Connection multiplexing: currently each ingest stream opens its own outbound TCP connection to the destination. Multiple streams between the same two peers should share one connection, with stream_id used to demultiplex frames. This is the priority/encapsulation scheme described in the architecture — high-priority and low-latency frames from different streams travel over the same socket rather than competing across separate sockets.