mikael-lovqvists-claude-agent/video-setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
49e5076eeab8eacabdc69212b9071046c7355cb4
video-setup/docs/protocol.md
mikael-lovqvists-claude-agent 49e5076eea Fix menu controls: wire up menu items in ctrl_enum_cb; document control commands 
src/node/main.c: ctrl_enum_cb was discarding menu_count and menu_items,
causing empty dropdowns for all MENU/INTEGER_MENU controls. Added a
menu item pool (MAX_MENU_POOL=128 items) to Ctrl_Build; the callback now
copies items into the pool and sets menu_count/menu_items on the control.

docs/protocol.md: add missing sections — str8 primitive, ENUM_DEVICES,
ENUM_CONTROLS (with control type/flag tables and menu item notes),
GET_CONTROL, and SET_CONTROL schemas.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 02:00:18 +00:00

394 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Protocol Reference
This document describes the full wire protocol used between nodes. All multi-byte integers are **little-endian**. Serialization primitives (`put_u16`, `get_u32`, etc.) are defined in [`serial.h`](../../include/serial.h).
---
## Layers
```mermaid
flowchart TD
A[TCP connection] --> B[Transport layer<br>frame header + length-prefixed payload]
B --> C[Message type dispatcher<br>routes payload to handler]
C --> D1[Video frame handler<br>stream_id + compressed data]
C --> D2[Control request/response handler<br>request_id + command fields]
C --> D3[Stream event handler<br>stream_id + event_code]
```
- **TCP** — reliable byte stream; provides ordering and delivery but no message boundaries
- **Transport layer** — adds a 6-byte header to every message, giving a length so any node can skip or forward unknown messages
- **Message type dispatcher** — reads `message_type` and routes the payload to the appropriate handler
- **Handlers** — interpret the payload according to their own schema; the transport layer has no knowledge of handler internals
---
## Transport Layer
### Frame Format
Every message on the wire is a frame:
```
+------------------+------------------------+--------------------+
| message_type: u16 | payload_length: u32 | payload: bytes ... |
+------------------+------------------------+--------------------+
2 bytes 4 bytes payload_length bytes
```
Total header size: **6 bytes**.
`payload_length` is the byte count of the payload only — it does not include the 6-byte header itself.
A node that does not recognise `message_type` can skip the frame by consuming exactly `payload_length` bytes and discarding them. This allows relays and future nodes to forward or ignore unknown message types without understanding their structure.
### Byte Order
All fields are little-endian. This applies to the header and to all fields within payloads.
### Connection Model
Each TCP connection carries a single logical channel between two peers. Multiple streams (video, control, events) are multiplexed on the same connection by `message_type` and by stream/request identifiers inside payloads. There is no connection-level stream identifier in the header — that information belongs to the payload.
---
## Message Types
| Value | Name | Description |
|---|---|---|
| `0x0001` | `VIDEO_FRAME` | One compressed video frame for a stream |
| `0x0002` | `CONTROL_REQUEST` | Request from one node to another |
| `0x0003` | `CONTROL_RESPONSE` | Response to a prior control request |
| `0x0004` | `STREAM_EVENT` | Lifecycle signal for a stream (interrupted, resumed) |
| `0x0010` | `DISCOVERY_ANNOUNCE` | UDP multicast node announcement (see [Discovery](#discovery)) |
Values not listed are reserved. A node receiving an unknown type must skip the payload (`payload_length` bytes) and continue reading.
---
## Payload Schemas
### `VIDEO_FRAME` (0x0001)
```
+---------------+----------------------------------------------+
| stream_id: u16 | frame data (compressed, codec per stream_open) |
+---------------+----------------------------------------------+
2 bytes payload_length - 2 bytes
```
`stream_id` identifies which video stream this frame belongs to. The codec is established at stream open time (see [Stream Lifecycle](#stream-lifecycle)) and does not appear in every frame.
### `CONTROL_REQUEST` (0x0002)
```
+------------------+----------------+---------------------------+
| request_id: u16 | command: u16 | command-specific fields |
+------------------+----------------+---------------------------+
2 bytes 2 bytes remaining bytes
```
`request_id` is chosen by the sender and echoed in the matching `CONTROL_RESPONSE`. It is used to correlate responses to requests when multiple requests are in flight simultaneously.
`command` values:
| Value | Command | Description |
|---|---|---|
| `0x0001` | `STREAM_OPEN` | Open a new video stream on this connection |
| `0x0002` | `STREAM_CLOSE` | Close a video stream |
| `0x0003` | `ENUM_DEVICES` | List V4L2 devices on the remote node |
| `0x0004` | `ENUM_CONTROLS` | List V4L2 controls for a device |
| `0x0005` | `GET_CONTROL` | Get a V4L2 control value |
| `0x0006` | `SET_CONTROL` | Set a V4L2 control value |
| `0x0007` | `ENUM_MONITORS` | List X11 monitors (XRandR) on the remote node |
### `CONTROL_RESPONSE` (0x0003)
```
+------------------+---------------+---------------------------+
| request_id: u16 | status: u16 | response-specific fields |
+------------------+---------------+---------------------------+
2 bytes 2 bytes remaining bytes
```
`request_id` matches the originating request. `status` values:
| Value | Meaning |
|---|---|
| `0x0000` | OK |
| `0x0001` | Error — generic failure |
| `0x0002` | Error — unknown command |
| `0x0003` | Error — invalid parameters |
| `0x0004` | Error — resource not found |
### `STREAM_EVENT` (0x0004)
```
+---------------+------------------+---------------------------+
| stream_id: u16 | event_code: u8 | event-specific fields |
+---------------+------------------+---------------------------+
2 bytes 1 byte remaining bytes
```
`event_code` values:
| Value | Name | Meaning |
|---|---|---|
| `0x01` | `STREAM_INTERRUPTED` | Device lost; frames will stop. Receiver should reset parser state and discard any partial frame. |
| `0x02` | `STREAM_RESUMED` | Device recovered; a clean frame follows. |
---
## Stream Lifecycle
Before video frames can flow, a stream must be opened. This establishes the codec and pixel format so receivers can decode frames without per-frame metadata.
### Opening a Stream (`STREAM_OPEN` request)
The sender issues a `CONTROL_REQUEST` with command `STREAM_OPEN`:
```
+------------------+------------------+-----------------+------------------+-------------------+----------------+
| request_id: u16 | command: u16 | stream_id: u16 | format: u16 | pixel_format: u16 | origin: u16 |
+------------------+------------------+-----------------+------------------+-------------------+----------------+
```
| Field | Description |
|---|---|
| `stream_id` | Chosen by the sender; identifies this stream on subsequent `VIDEO_FRAME` and `STREAM_EVENT` messages |
| `format` | Wire format of the compressed frame data (see [Codec Formats](#codec-formats)) |
| `pixel_format` | Pixel layout for raw formats; zero for compressed formats (see [Pixel Formats](#pixel-formats)) |
| `origin` | How the frames were produced; informational only, does not affect decoding (see [Origins](#origins)) |
The receiver responds with `CONTROL_RESPONSE`. On `status = OK` the stream is open and `VIDEO_FRAME` messages with that `stream_id` may follow immediately.
### Closing a Stream (`STREAM_CLOSE` request)
```
+------------------+------------------+-----------------+
| request_id: u16 | command: u16 | stream_id: u16 |
+------------------+------------------+-----------------+
```
After a close response, no further `VIDEO_FRAME` messages should be sent for that `stream_id`.
---
## Codec Formats
Carried in the `format` field of `STREAM_OPEN`.
| Value | Format |
|---|---|
| `0x0001` | MJPEG |
| `0x0002` | H.264 |
| `0x0003` | H.265 / HEVC |
| `0x0004` | AV1 |
| `0x0005` | FFV1 |
| `0x0006` | ProRes |
| `0x0007` | QOI |
| `0x0008` | Raw pixels (requires `pixel_format`) |
| `0x0009` | Raw pixels + ZSTD (requires `pixel_format`) |
## Pixel Formats
Carried in the `pixel_format` field of `STREAM_OPEN`. Zero and ignored for compressed formats.
| Value | Layout |
|---|---|
| `0x0001` | BGRA 8:8:8:8 |
| `0x0002` | RGBA 8:8:8:8 |
| `0x0003` | BGR 8:8:8 |
| `0x0004` | YUV 4:2:0 planar |
| `0x0005` | YUV 4:2:2 packed |
## Origins
Carried in the `origin` field of `STREAM_OPEN`. Informational — does not affect decoding.
| Value | Origin |
|---|---|
| `0x0001` | Device native (camera or capture card encoded it directly) |
| `0x0002` | libjpeg-turbo |
| `0x0003` | ffmpeg (libavcodec) |
| `0x0004` | ffmpeg (subprocess) |
| `0x0005` | VA-API direct |
| `0x0006` | NVENC direct |
| `0x0007` | Software (other) |
---
## Discovery
Node discovery uses UDP multicast. The wire format is a standard transport frame (same 6-byte header) sent to the multicast group rather than a TCP peer.
### Announcement Frame (`DISCOVERY_ANNOUNCE`, 0x0010)
Sent periodically by every node and immediately on startup.
```
+----------------------+------------+-----------------+------------------+-------------------+------------------+
| protocol_version: u8 | site_id: u16 | tcp_port: u16 | function_flags: u16 | name_len: u8 | name: bytes |
+----------------------+------------+-----------------+------------------+-------------------+------------------+
```
| Field | Description |
|---|---|
| `protocol_version` | Wire format version; currently `1` |
| `site_id` | Site this node belongs to; `0` = local / unassigned |
| `tcp_port` | Port where this node accepts transport connections |
| `function_flags` | Bitfield of node capabilities (see below) |
| `name_len` | Byte length of the following name string |
| `name` | Node name in `namespace:instance` form, e.g. `v4l2:microscope` |
`function_flags` bits:
| Bit | Mask | Role |
|---|---|---|
| 0 | `0x0001` | Source — produces video |
| 1 | `0x0002` | Relay — receives and distributes streams |
| 2 | `0x0004` | Sink — consumes video |
| 3 | `0x0008` | Controller — has a user-facing control interface |
A node may set multiple bits.
### Multicast Parameters
| Parameter | Value |
|---|---|
| Group | `224.0.0.251` |
| Port | `5353` |
| TTL | `1` (LAN only) |
No Avahi or Bonjour dependency — nodes open a raw UDP multicast socket directly using standard POSIX APIs.
### Site ID Translation
`site_id = 0` means "local / unassigned". Both sides of a site-to-site link will independently default to `0`, so a gateway cannot forward announcements across the boundary without rewriting the field — all nodes on both sides would appear identical.
The gateway assigns a distinct non-zero `site_id` to each side and rewrites `site_id` in all announcements (and any protocol messages carrying a `site_id`) as they cross the boundary. Receiving an announcement with `site_id = 0` on a cross-site link is a gateway protocol error.
---
## Serialisation Primitives
`str8` — a length-prefixed UTF-8 string, not NUL-terminated on the wire:
```
+------------+-------------------+
| length: u8 | bytes: length × u8 |
+------------+-------------------+
```
Maximum string length is 255 bytes.
---
## Control Commands
All requests follow the `CONTROL_REQUEST` frame format (request_id + command + command-specific fields). All responses follow the `CONTROL_RESPONSE` frame format (request_id + status + response-specific fields).
### `ENUM_DEVICES` (0x0003)
**Request** — no extra fields beyond request_id and command.
**Response** on status `OK`:
```
+-------------------+
| media_count: u16 |
+-------------------+
× media_count:
+------------+------------+-----------+-------------+-----------------+
| path: str8 | driver:str8| model:str8| bus_info:str8| vnode_count: u8 |
+------------+------------+-----------+-------------+-----------------+
× vnode_count:
+------------+-----------------+------------------+-------------------+------------------+----------------+------------------+
| path: str8 | entity_name:str8| entity_type: u32 | entity_flags: u32 | device_caps: u32 | pad_flags: u8 | is_capture: u8 |
+------------+-----------------+------------------+-------------------+------------------+----------------+------------------+
+----------------------+
| standalone_count: u16 |
+----------------------+
× standalone_count:
+------------+------------+
| path: str8 | name: str8 |
+------------+------------+
```
`device_caps` carries `V4L2_CAP_*` bits from `VIDIOC_QUERYCAP` (using `device_caps` if `V4L2_CAP_DEVICE_CAPS` is set, otherwise `capabilities`). Notable bits: `0x00000001` = `VIDEO_CAPTURE`, `0x00800000` = `META_CAPTURE`.
`is_capture` is `1` if `device_caps & V4L2_CAP_VIDEO_CAPTURE`, else `0`.
### `ENUM_CONTROLS` (0x0004)
**Request**:
```
+--------------------+
| device_index: u16 |
+--------------------+
```
**Response** on status `OK`:
```
+---------------+
| count: u16 |
+---------------+
× count:
+---------+---------+----------+------------+----------+---------+----------+-------------+-----------------+------------------+
| id: u32 | type: u8 | flags: u32 | name: str8 | min: i32 | max: i32 | step: i32 | default: i32 | current: i32 | menu_count: u8 |
+---------+---------+----------+------------+----------+---------+----------+-------------+-----------------+------------------+
× menu_count:
+------------+------------+------------------+
| index: u32 | name: str8 | int_value: i64 |
+------------+------------+------------------+
```
`type` values match `V4L2_CTRL_TYPE_*`:
| Value | Type |
|---|---|
| `1` | `INTEGER` — use a slider |
| `2` | `BOOLEAN` — use a checkbox |
| `3` | `MENU` — use a select; `name` of each menu item is the label |
| `4` | `BUTTON` — use a button |
| `9` | `INTEGER_MENU` — use a select; `int_value` of each menu item is the label |
`flags` bits (from `V4L2_CTRL_FLAG_*`): `0x0001` = disabled, `0x0002` = grabbed (read-only due to another active control), `0x0004` = read-only.
For `MENU` and `INTEGER_MENU` controls, set the control value to a menu item's `index` (not its `int_value`). `int_value` is informational display text only.
Menu items may have non-contiguous `index` values (gaps where the driver returns `EINVAL` for `VIDIOC_QUERYMENU`).
### `GET_CONTROL` (0x0005)
**Request**:
```
+--------------------+------------------+
| device_index: u16 | control_id: u32 |
+--------------------+------------------+
```
**Response** on status `OK`:
```
+-------------+
| value: i32 |
+-------------+
```
### `SET_CONTROL` (0x0006)
**Request**:
```
+--------------------+------------------+-------------+
| device_index: u16 | control_id: u32 | value: i32 |
+--------------------+------------------+-------------+
```
**Response** — no extra fields beyond request_id and status.
For `MENU` and `INTEGER_MENU` controls, `value` must be a valid menu item `index` as returned by `ENUM_CONTROLS`.
Reference in New Issue View Git Blame Copy Permalink