From f5764940e6d276382848ba247c738e4e9541bfe8 Mon Sep 17 00:00:00 2001 From: mikael-lovqvists-claude-agent Date: Sun, 29 Mar 2026 08:03:30 +0000 Subject: [PATCH] Docs: display sink commands, GLFW multi-window notes, planning updates - protocol.md: add START_DISPLAY (0x000A) and STOP_DISPLAY (0x000B) wire schemas and field descriptions; add both to command table - xorg.md: add 'Multiple windows' section covering glfwPollEvents global behaviour, per-context glfwMakeContextCurrent requirement, and glfwInit/glfwTerminate ref-counting; includes the gotcha that short-circuiting the event loop can starve non-polled windows - planning.md: add cooperative capture release deferred decision; add xorg viewer remote controls (zoom, pan, scale, future shader post-processing) to deferred decisions; note xorg viewer controls not yet exposed remotely in module table Co-Authored-By: Claude Sonnet 4.6 --- docs/protocol.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++++ docs/xorg.md | 10 ++++++++++ planning.md | 4 +++- 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/protocol.md b/docs/protocol.md index 4e973b4..d49b607 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -101,6 +101,8 @@ packet-beta | `0x0007` | `ENUM_MONITORS` | List X11 monitors (XRandR) on the remote node | | `0x0008` | `START_INGEST` | Set wanted state: open V4L2 device, connect outbound, begin streaming | | `0x0009` | `STOP_INGEST` | Set wanted state: stop ingest stream and disconnect | +| `0x000A` | `START_DISPLAY` | Open a viewer window on the sink node and display incoming frames for the given stream | +| `0x000B` | `STOP_DISPLAY` | Close the viewer window for the given stream | ### `CONTROL_RESPONSE` (0x0003) @@ -470,3 +472,50 @@ packet-beta ``` **Response** — no extra fields beyond request_id and status. + +### `START_DISPLAY` (0x000A) + +Opens a viewer window on a sink node and routes incoming `VIDEO_FRAME` messages for `stream_id` to it. + +**Request**: + +```mermaid +%%{init: {'packet': {'bitsPerRow': 16}}}%% +packet-beta +0-15: "request_id" +16-31: "command = 0x000A" +32-47: "stream_id" +48-63: "win_x (i16)" +64-79: "win_y (i16)" +80-95: "win_w" +96-111: "win_h" +112-119: "scale" +120-127: "anchor" +``` + +| Field | Description | +|---|---| +| `stream_id` | Stream to display; must match incoming `VIDEO_FRAME` stream_id | +| `win_x`, `win_y` | Window screen position (signed; for multi-monitor placement) | +| `win_w`, `win_h` | Window size in pixels; `0` = default (1280×720) | +| `scale` | `0`=stretch `1`=fit `2`=fill `3`=1:1 | +| `anchor` | `0`=center `1`=topleft | + +**Response** — no extra fields beyond request_id and status. `OK` means the display slot was reserved; the window opens asynchronously on the main thread. + +### `STOP_DISPLAY` (0x000B) + +Closes the viewer window for the given stream. + +**Request**: + +```mermaid +%%{init: {'packet': {'bitsPerRow': 16}}}%% +packet-beta +0-15: "request_id" +16-31: "command = 0x000B" +32-47: "stream_id" +``` + +**Response** — no extra fields beyond request_id and status. + diff --git a/docs/xorg.md b/docs/xorg.md index dd227de..c4b1c86 100644 --- a/docs/xorg.md +++ b/docs/xorg.md @@ -107,6 +107,16 @@ A `framebuffer_size_callback` registered on the window calls `render()` synchron Threading note: the GL context must be used from the thread that created it. In the video node, incoming frames arrive on a network receive thread. A frame queue between the receive thread and the render thread (which owns the GL context) is the correct model — the render thread drains the queue each poll iteration rather than having the network thread call push functions directly. +### Multiple windows + +GLFW supports multiple windows from the same thread. `glfwCreateWindow` can be called repeatedly; each call returns an independent window handle with its own GL context. The video node uses this to display several streams simultaneously (one window per active `Display_Slot`). + +**`glfwPollEvents` is global.** It drains the event queue for all windows at once, not just the one associated with the viewer it is called through. When iterating over multiple display slots and calling `xorg_viewer_handle_events` on each, only the first call does real work; subsequent calls are no-ops because the queue is already empty. This is harmless but worth knowing: if the loop is ever restructured so that event polling is conditional or short-circuited, all windows need at least one `glfwPollEvents` call per iteration or they will stop responding to input. + +**Each window has its own GL context.** `glfwMakeContextCurrent` must be called before any GL operations to ensure calls go to the right context. The push functions (`push_yuv420`, `push_bgra`, `push_mjpeg`) and `poll` do this automatically. Code that calls GL functions directly must make the correct context current first. + +**`glfwInit`/`glfwTerminate` are ref-counted** in the xorg module. The first `xorg_viewer_open` call initialises GLFW; `glfwTerminate` is deferred until the last viewer is closed. Do not call `glfwTerminate` directly — use `xorg_viewer_close` and let the ref count manage it. + ### Renderer: Vulkan (future alternative) A Vulkan renderer is planned as an alternative to the OpenGL one. GLFW's surface creation API is renderer-agnostic, so the window management and input handling code is shared. Only the renderer backend changes. diff --git a/planning.md b/planning.md index 515d220..1afd0cb 100644 --- a/planning.md +++ b/planning.md @@ -57,7 +57,7 @@ Modules are listed in intended build order. Each depends only on modules above i | 7 | `protocol` | done | Typed `write_*`/`read_*` functions for all message types; builds on serial + transport | | — | `node` | done | Video node binary — config, discovery, transport server, V4L2/media control request handlers | | 8 | `test_image` | done | Test pattern generator — colour bars, luminance ramp, grid crosshatch; YUV420/BGRA output | -| 9 | `xorg` | done | GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented | +| 9 | `xorg` | done | GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented; viewer controls (zoom, pan, scale policy) not yet exposed remotely | | 10 | `reconciler` | done | Generic wanted/current state machine reconciler — resource state graphs, BFS pathfinding, event + periodic tick; used by node to manage V4L2 devices, transport connections, and future resources (codec processes etc.) | | 11 | `frame_alloc` | not started | Per-frame allocation with bookkeeping (byte budget, ref counting) | | 12 | `relay` | not started | Input dispatch to output queues (low-latency and completeness modes) | @@ -113,3 +113,5 @@ These are open questions tracked in `architecture.md` that do not need to be res - Transport for relay edges (TCP / UDP / shared memory) - Node discovery mechanism - Hard vs soft byte budget limits +- Cooperative capture release: if a capture source has no live downstream targets for a configurable time window, stop capture and release the device. Intended as a resource-conservation policy rather than an immediate reaction to disconnect events. Requires the node to track downstream liveness (e.g. last successful send timestamp per output) and implement a reaper timer. +- Xorg viewer remote controls: expose viewer state (zoom, pan, scale policy, anchor) as enumerable/settable controls via the protocol, analogous to V4L2 controls. Future extension: shader-based post-processing — initial candidates are a colour-correction shader and custom user-provided GLSL fragment shaders sent over the wire.