mikael-lovqvists-claude-agent/video-setup

Files

mikael-lovqvists-claude-agent 961933e714 Docs: add peer addressing and connection multiplexing to deferred decisions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-03-29 19:36:13 +00:00

9.7 KiB

Raw Blame History

Planning

Approach

Build the system module by module in C11. Each module is a translation unit (.h + .c) with a clearly defined API. Modules are exercised by small driver programs in dev/ before anything depends on them. This keeps each unit independently testable and prevents architectural decisions from being made prematurely in code.

The final binary is a single configurable node program. That integration work comes after the modules are solid.

Directory Structure

video-setup/
  src/
    modules/
      common/        - shared definitions (error types, base types)
      config/        - INI file loader with schema-driven defaults, typed getters
      media_ctrl/    - Linux Media Controller API (topology, pad formats, links)
      v4l2_ctrl/     - V4L2 camera controls (enumerate, get, set)
      serial/        - little-endian binary serialization primitives
      transport/     - framed TCP stream, single-write send
      discovery/     - UDP multicast announcements, peer table, found/lost callbacks
      protocol/      - typed write_*/read_* message functions
      test_image/    - test pattern generator (colour bars, ramp, grid; YUV420/BGRA)
      xorg/          - GLFW+OpenGL viewer sink; stub for headless builds
      reconciler/    - generic wanted/current state machine reconciler
      ingest/        - V4L2 capture loop, MMAP buffers, on_frame callback
    node/            - video node binary (source + display sink roles)
  include/           - public headers
  dev/
    cli/             - exploratory CLI drivers, one per module
    web/             - development web UI (Node.js/Express); browser-side equivalent
                       of the CLI tools; depends on protocol being finalised
    experiments/     - freeform experiments
  tools/
    gen_font_atlas/  - build-time bitmap font atlas generator (Python/Pillow);
                       outputs build/gen/font_atlas.h consumed by xorg module
  tests/             - automated tests (later)
  Makefile
  architecture.md
  planning.md
  conventions.md

Module Order

Modules are listed in intended build order. Each depends only on modules above it.

#	Module	Status	Notes
1	`common`	done	Error types, base definitions — no dependencies
—	`config`	done	INI file loader with schema-driven defaults, typed getters, FLAGS type for bitmask values
2	`media_ctrl`	done	Media Controller API — device and topology enumeration, pad format config
3	`v4l2_ctrl`	done	V4L2 controls — enumerate, get, set camera parameters
4	`serial`	done	`put`/`get` primitives for little-endian binary serialization into byte buffers
5	`transport`	done	Encapsulated transport — frame header, TCP stream abstraction, single-write send
6	`discovery`	done	UDP multicast announcements, peer table, found/lost callbacks
7	`protocol`	done	Typed `write_`/`read_` functions for all message types; builds on serial + transport
8	`test_image`	done	Test pattern generator — colour bars, luminance ramp, grid crosshatch; YUV420/BGRA output
9	`xorg`	done	GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented; viewer controls (zoom, pan, scale policy) not yet exposed remotely
10	`reconciler`	done	Generic wanted/current state machine reconciler — resource state graphs, BFS pathfinding, event + periodic tick; used by node to manage V4L2 devices, transport connections, and future resources (codec processes etc.)
11	`ingest`	done	V4L2 capture loop — open device, negotiate MJPEG format, MMAP buffers, capture thread with on_frame callback; start/stop lifecycle managed by reconciler
—	`node`	done	Video node binary — config, discovery, transport server, V4L2/media control request handlers; display sink role (START_DISPLAY/STOP_DISPLAY handlers, multi-window xorg viewer, declarative display slot reconciler)
12	`frame_alloc`	not started	Per-frame allocation with bookkeeping (byte budget, ref counting)
13	`relay`	not started	Input dispatch to output queues (low-latency and completeness modes)
14	`archive`	not started	Write frames to disk, control messages to binary log
15	`codec`	not started	Per-frame encode/decode — MJPEG (libjpeg-turbo), QOI, ZSTD-raw, VA-API H.264 intra; used by screen grab source and archive
16	`web node`	not started	Node.js/Express peer — speaks binary protocol on socket side, HTTP/WebSocket to browser; `protocol.mjs` mirrors C protocol module
—	`mjpeg_scan`	future	EOI marker scanner for misbehaving hardware that does not deliver clean per-buffer frames; not part of the primary pipeline

Dev Tools

CLI (`dev/cli/`)

Each module gets a corresponding CLI driver that exercises its API and serves as both an integration check and a useful development tool.

Driver	Exercises	Notes
`media_ctrl_cli`	`media_ctrl`	List media devices, show topology, configure pad formats
`v4l2_ctrl_cli`	`v4l2_ctrl`	List controls, get/set values — lightweight `v4l2-ctl` equivalent
`transport_cli`	`transport`	Send/receive framed messages, inspect headers
`test_image_cli`	`test_image`	Generate test patterns, write PPM for visual inspection
`xorg_cli`	`xorg`	Display test pattern in viewer window; exercises scale/anchor modes and text overlays
`v4l2_view_cli`	V4L2 + `xorg`	Live camera viewer — auto-selects highest-FPS format, FPS/format overlay; bypasses node system
`stream_send_cli`	V4L2 + `transport` + `protocol`	Capture MJPEG from V4L2, connect to receiver, send VIDEO_FRAME messages; prints fps/Mbps stats
`stream_recv_cli`	`transport` + `protocol` + `xorg`	Listen for incoming VIDEO_FRAME stream, display in viewer; fps/Mbps overlay; threaded transport→GL handoff
`reconciler_cli`	`reconciler`	Simulated state machine experiment — define resources with fake transitions, drive reconciler via CLI commands; validates the generic reconciler before wiring into the node
`controller_cli`	`transport` + `protocol`	Interactive controller REPL — connects to a running node by host:port; supports enum-devices, enum-controls, get/set-control, start-ingest, stop-ingest, start-display, stop-display

Web UI (`dev/web/`)

A Node.js/Express development web UI that connects to running video nodes as a binary protocol peer. Exposes V4L2 control inspection and adjustment, media topology view, and stream state through a browser interface. Supports live peer discovery via SSE — discovered nodes appear automatically in the UI.

This is a development aid, not the production dashboard. The production dashboard (full stream configuration UI) is a later, separate project.

Future: Protocol Preprocessor

The C protocol module and JavaScript protocol.mjs will eventually be generated from a single schema by a future preprocessor. This eliminates drift between the two implementations. The preprocessor also handles error location codes (see common/error). Neither the schema format nor the preprocessor tool exists yet — the hand-written implementations are the interim state.

Deferred Decisions

These are open questions tracked in architecture.md that do not need to be resolved before module work begins:

Graph representation format
Connection establishment model (push vs pull)
Completeness queue drop policy (oldest vs newest, per-output config)
Stream ID remapping across relay hops
Transport for relay edges (TCP / UDP / shared memory)
Node discovery mechanism
Hard vs soft byte budget limits
Cooperative capture release: if a capture source has no live downstream targets for a configurable time window, stop capture and release the device. Intended as a resource-conservation policy rather than an immediate reaction to disconnect events. Requires the node to track downstream liveness (e.g. last successful send timestamp per output) and implement a reaper timer.
Unified device model: active display windows should be registered as devices alongside V4L2 cameras, using the same ENUM_DEVICES / ENUM_CONTROLS / GET_CONTROL / SET_CONTROL protocol. START_DISPLAY would return a device_id for the opened window; controls (scale, anchor, position, size, zoom, pan) are then addressable as (device_id, control_id) pairs like any other device. Requires a device_type field in ENUM_DEVICES responses so controllers can distinguish V4L2 devices from display windows. Future device types: codec processes, screen grab sources. This extends naturally to shader-based post-processing and other viewer state as controls.
controller_cli improvements: (1) readline for line editing and history; (2) discovery integration — run a discovery listener, expose a peers command listing discovered nodes, connect <id|host:port> to switch target, auto-print newly discovered nodes.
start-ingest peer addressing: the dest_host + dest_port in START_INGEST is awkward to type manually and requires the caller to know the target's TCP port. Should accept a peer ID (index from the discovered peer table on the node) so the node can resolve the address itself. Requires the node to run discovery and expose its peer table.
Connection multiplexing: currently each ingest stream opens its own outbound TCP connection to the destination. Multiple streams between the same two peers should share one connection, with stream_id used to demultiplex frames. This is the priority/encapsulation scheme described in the architecture — high-priority and low-latency frames from different streams travel over the same socket rather than competing across separate sockets.

9.7 KiB Raw Blame History