Docs: display sink commands, GLFW multi-window notes, planning updates

- protocol.md: add START_DISPLAY (0x000A) and STOP_DISPLAY (0x000B) wire
  schemas and field descriptions; add both to command table
- xorg.md: add 'Multiple windows' section covering glfwPollEvents global
  behaviour, per-context glfwMakeContextCurrent requirement, and
  glfwInit/glfwTerminate ref-counting; includes the gotcha that
  short-circuiting the event loop can starve non-polled windows
- planning.md: add cooperative capture release deferred decision;
  add xorg viewer remote controls (zoom, pan, scale, future shader
  post-processing) to deferred decisions; note xorg viewer controls
  not yet exposed remotely in module table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-29 08:03:30 +00:00
parent 32d31cbd1e
commit f5764940e6
3 changed files with 62 additions and 1 deletions

View File

@@ -101,6 +101,8 @@ packet-beta
| `0x0007` | `ENUM_MONITORS` | List X11 monitors (XRandR) on the remote node |
| `0x0008` | `START_INGEST` | Set wanted state: open V4L2 device, connect outbound, begin streaming |
| `0x0009` | `STOP_INGEST` | Set wanted state: stop ingest stream and disconnect |
| `0x000A` | `START_DISPLAY` | Open a viewer window on the sink node and display incoming frames for the given stream |
| `0x000B` | `STOP_DISPLAY` | Close the viewer window for the given stream |
### `CONTROL_RESPONSE` (0x0003)
@@ -470,3 +472,50 @@ packet-beta
```
**Response** — no extra fields beyond request_id and status.
### `START_DISPLAY` (0x000A)
Opens a viewer window on a sink node and routes incoming `VIDEO_FRAME` messages for `stream_id` to it.
**Request**:
```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x000A"
32-47: "stream_id"
48-63: "win_x (i16)"
64-79: "win_y (i16)"
80-95: "win_w"
96-111: "win_h"
112-119: "scale"
120-127: "anchor"
```
| Field | Description |
|---|---|
| `stream_id` | Stream to display; must match incoming `VIDEO_FRAME` stream_id |
| `win_x`, `win_y` | Window screen position (signed; for multi-monitor placement) |
| `win_w`, `win_h` | Window size in pixels; `0` = default (1280×720) |
| `scale` | `0`=stretch `1`=fit `2`=fill `3`=1:1 |
| `anchor` | `0`=center `1`=topleft |
**Response** — no extra fields beyond request_id and status. `OK` means the display slot was reserved; the window opens asynchronously on the main thread.
### `STOP_DISPLAY` (0x000B)
Closes the viewer window for the given stream.
**Request**:
```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x000B"
32-47: "stream_id"
```
**Response** — no extra fields beyond request_id and status.

View File

@@ -107,6 +107,16 @@ A `framebuffer_size_callback` registered on the window calls `render()` synchron
Threading note: the GL context must be used from the thread that created it. In the video node, incoming frames arrive on a network receive thread. A frame queue between the receive thread and the render thread (which owns the GL context) is the correct model — the render thread drains the queue each poll iteration rather than having the network thread call push functions directly.
### Multiple windows
GLFW supports multiple windows from the same thread. `glfwCreateWindow` can be called repeatedly; each call returns an independent window handle with its own GL context. The video node uses this to display several streams simultaneously (one window per active `Display_Slot`).
**`glfwPollEvents` is global.** It drains the event queue for all windows at once, not just the one associated with the viewer it is called through. When iterating over multiple display slots and calling `xorg_viewer_handle_events` on each, only the first call does real work; subsequent calls are no-ops because the queue is already empty. This is harmless but worth knowing: if the loop is ever restructured so that event polling is conditional or short-circuited, all windows need at least one `glfwPollEvents` call per iteration or they will stop responding to input.
**Each window has its own GL context.** `glfwMakeContextCurrent` must be called before any GL operations to ensure calls go to the right context. The push functions (`push_yuv420`, `push_bgra`, `push_mjpeg`) and `poll` do this automatically. Code that calls GL functions directly must make the correct context current first.
**`glfwInit`/`glfwTerminate` are ref-counted** in the xorg module. The first `xorg_viewer_open` call initialises GLFW; `glfwTerminate` is deferred until the last viewer is closed. Do not call `glfwTerminate` directly — use `xorg_viewer_close` and let the ref count manage it.
### Renderer: Vulkan (future alternative)
A Vulkan renderer is planned as an alternative to the OpenGL one. GLFW's surface creation API is renderer-agnostic, so the window management and input handling code is shared. Only the renderer backend changes.

View File

@@ -57,7 +57,7 @@ Modules are listed in intended build order. Each depends only on modules above i
| 7 | `protocol` | done | Typed `write_*`/`read_*` functions for all message types; builds on serial + transport |
| — | `node` | done | Video node binary — config, discovery, transport server, V4L2/media control request handlers |
| 8 | `test_image` | done | Test pattern generator — colour bars, luminance ramp, grid crosshatch; YUV420/BGRA output |
| 9 | `xorg` | done | GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented |
| 9 | `xorg` | done | GLFW+OpenGL viewer sink — YUV420/BGRA/MJPEG display, all scale/anchor modes, bitmap font atlas text overlays; XRandR queries and screen grab not yet implemented; viewer controls (zoom, pan, scale policy) not yet exposed remotely |
| 10 | `reconciler` | done | Generic wanted/current state machine reconciler — resource state graphs, BFS pathfinding, event + periodic tick; used by node to manage V4L2 devices, transport connections, and future resources (codec processes etc.) |
| 11 | `frame_alloc` | not started | Per-frame allocation with bookkeeping (byte budget, ref counting) |
| 12 | `relay` | not started | Input dispatch to output queues (low-latency and completeness modes) |
@@ -113,3 +113,5 @@ These are open questions tracked in `architecture.md` that do not need to be res
- Transport for relay edges (TCP / UDP / shared memory)
- Node discovery mechanism
- Hard vs soft byte budget limits
- Cooperative capture release: if a capture source has no live downstream targets for a configurable time window, stop capture and release the device. Intended as a resource-conservation policy rather than an immediate reaction to disconnect events. Requires the node to track downstream liveness (e.g. last successful send timestamp per output) and implement a reaper timer.
- Xorg viewer remote controls: expose viewer state (zoom, pan, scale policy, anchor) as enumerable/settable controls via the protocol, analogous to V4L2 controls. Future extension: shader-based post-processing — initial candidates are a colour-correction shader and custom user-provided GLSL fragment shaders sent over the wire.