video-setup/docs/protocol.md

# Protocol Reference

This document describes the full wire protocol used between nodes. All multi-byte integers are **little-endian**. Serialization primitives (`put_u16`, `get_u32`, etc.) are defined in [`serial.h`](../../include/serial.h).

---

## Layers

```mermaid
flowchart TD
    A[TCP connection] --> B[Transport layer<br>frame header + length-prefixed payload]
    B --> C[Message type dispatcher<br>routes payload to handler]
    C --> D1[Video frame handler<br>stream_id + compressed data]
    C --> D2[Control request/response handler<br>request_id + command fields]
    C --> D3[Stream event handler<br>stream_id + event_code]
```

- **TCP** — reliable byte stream; provides ordering and delivery but no message boundaries
- **Transport layer** — adds a 6-byte header to every message, giving a length so any node can skip or forward unknown messages
- **Message type dispatcher** — reads `message_type` and routes the payload to the appropriate handler
- **Handlers** — interpret the payload according to their own schema; the transport layer has no knowledge of handler internals

---

## Transport Layer

### Frame Format

Every message on the wire is a frame:

```mermaid
%%{init: {'packet': {'bitsPerRow': 48}}}%%
packet-beta
0-15: "message_type"
16-47: "payload_length"
48-95: "payload …"
```

Total header size: **6 bytes**. `payload_length` is the byte count of the payload only — it does not include the 6-byte header itself.

A node that does not recognise `message_type` can skip the frame by consuming exactly `payload_length` bytes and discarding them. This allows relays and future nodes to forward or ignore unknown message types without understanding their structure.

### Byte Order

All fields are little-endian. This applies to the header and to all fields within payloads.

### Connection Model

Each TCP connection carries a single logical channel between two peers. Multiple streams (video, control, events) are multiplexed on the same connection by `message_type` and by stream/request identifiers inside payloads. There is no connection-level stream identifier in the header — that information belongs to the payload.

---

## Message Types

| Value | Name | Description |
|---|---|---|
| `0x0001` | `VIDEO_FRAME` | One compressed video frame for a stream |
| `0x0002` | `CONTROL_REQUEST` | Request from one node to another |
| `0x0003` | `CONTROL_RESPONSE` | Response to a prior control request |
| `0x0004` | `STREAM_EVENT` | Lifecycle signal for a stream (interrupted, resumed) |
| `0x0010` | `DISCOVERY_ANNOUNCE` | UDP multicast node announcement (see [Discovery](#discovery)) |

Values not listed are reserved. A node receiving an unknown type must skip the payload (`payload_length` bytes) and continue reading.

---

## Payload Schemas

### `VIDEO_FRAME` (0x0001)

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "stream_id"
16-31: "frame data …"
```

`stream_id` identifies which video stream this frame belongs to. The codec is established at stream open time (see [Stream Lifecycle](#stream-lifecycle)) and does not appear in every frame.

### `CONTROL_REQUEST` (0x0002)

```mermaid
packet-beta
0-15: "request_id"
16-31: "command"
32-63: "command-specific …"
```

`request_id` is chosen by the sender and echoed in the matching `CONTROL_RESPONSE`. It is used to correlate responses to requests when multiple requests are in flight simultaneously.

`command` values:

| Value | Command | Description |
|---|---|---|
| `0x0001` | `STREAM_OPEN` | Open a new video stream on this connection |
| `0x0002` | `STREAM_CLOSE` | Close a video stream |
| `0x0003` | `ENUM_DEVICES` | List V4L2 devices on the remote node |
| `0x0004` | `ENUM_CONTROLS` | List V4L2 controls for a device |
| `0x0005` | `GET_CONTROL` | Get a V4L2 control value |
| `0x0006` | `SET_CONTROL` | Set a V4L2 control value |
| `0x0007` | `ENUM_MONITORS` | List X11 monitors (XRandR) on the remote node |
| `0x0008` | `START_INGEST` | Set wanted state: open V4L2 device, connect outbound, begin streaming |
| `0x0009` | `STOP_INGEST` | Set wanted state: stop ingest stream and disconnect |
| `0x000A` | `START_DISPLAY` | Open a viewer window on the sink node and display incoming frames for the given stream |
| `0x000B` | `STOP_DISPLAY` | Close the viewer window for the given stream |

### `CONTROL_RESPONSE` (0x0003)

```mermaid
packet-beta
0-15: "request_id"
16-31: "status"
32-63: "response-specific …"
```

`request_id` matches the originating request. `status` values:

| Value | Meaning |
|---|---|
| `0x0000` | OK |
| `0x0001` | Error — generic failure |
| `0x0002` | Error — unknown command |
| `0x0003` | Error — invalid parameters |
| `0x0004` | Error — resource not found |

### `STREAM_EVENT` (0x0004)

```mermaid
%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-15: "stream_id"
16-23: "event_code"
24-55: "event-specific …"
```

`event_code` values:

| Value | Name | Meaning |
|---|---|---|
| `0x01` | `STREAM_INTERRUPTED` | Device lost; frames will stop. Receiver should reset parser state and discard any partial frame. |
| `0x02` | `STREAM_RESUMED` | Device recovered; a clean frame follows. |

---

## Stream Lifecycle

Before video frames can flow, a stream must be opened. This establishes the codec and pixel format so receivers can decode frames without per-frame metadata.

### Opening a Stream (`STREAM_OPEN` request)

```mermaid
packet-beta
0-15: "request_id"
16-31: "command = 0x0001"
32-47: "stream_id"
48-63: "format"
64-79: "pixel_format"
80-95: "origin"
```

| Field | Description |
|---|---|
| `stream_id` | Chosen by the sender; identifies this stream on subsequent `VIDEO_FRAME` and `STREAM_EVENT` messages |
| `format` | Wire format of the compressed frame data (see [Codec Formats](#codec-formats)) |
| `pixel_format` | Pixel layout for raw formats; zero for compressed formats (see [Pixel Formats](#pixel-formats)) |
| `origin` | How the frames were produced; informational only, does not affect decoding (see [Origins](#origins)) |

The receiver responds with `CONTROL_RESPONSE`. On `status = OK` the stream is open and `VIDEO_FRAME` messages with that `stream_id` may follow immediately.

### Closing a Stream (`STREAM_CLOSE` request)

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0002"
32-47: "stream_id"
```

After a close response, no further `VIDEO_FRAME` messages should be sent for that `stream_id`.

---

## Codec Formats

Carried in the `format` field of `STREAM_OPEN`.

| Value | Format |
|---|---|
| `0x0001` | MJPEG |
| `0x0002` | H.264 |
| `0x0003` | H.265 / HEVC |
| `0x0004` | AV1 |
| `0x0005` | FFV1 |
| `0x0006` | ProRes |
| `0x0007` | QOI |
| `0x0008` | Raw pixels (requires `pixel_format`) |
| `0x0009` | Raw pixels + ZSTD (requires `pixel_format`) |

## Pixel Formats

Carried in the `pixel_format` field of `STREAM_OPEN`. Zero and ignored for compressed formats.

| Value | Layout |
|---|---|
| `0x0001` | BGRA 8:8:8:8 |
| `0x0002` | RGBA 8:8:8:8 |
| `0x0003` | BGR 8:8:8 |
| `0x0004` | YUV 4:2:0 planar |
| `0x0005` | YUV 4:2:2 packed |

## Origins

Carried in the `origin` field of `STREAM_OPEN`. Informational — does not affect decoding.

| Value | Origin |
|---|---|
| `0x0001` | Device native (camera or capture card encoded it directly) |
| `0x0002` | libjpeg-turbo |
| `0x0003` | ffmpeg (libavcodec) |
| `0x0004` | ffmpeg (subprocess) |
| `0x0005` | VA-API direct |
| `0x0006` | NVENC direct |
| `0x0007` | Software (other) |

---

## Discovery

Node discovery uses UDP multicast. The wire format is a standard transport frame (same 6-byte header) sent to the multicast group rather than a TCP peer.

### Announcement Frame (`DISCOVERY_ANNOUNCE`, 0x0010)

Sent periodically by every node and immediately on startup.

```mermaid
%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-7: "protocol_version"
8-23: "site_id"
24-39: "tcp_port"
40-55: "function_flags"
56-63: "name_len"
64-95: "name …"
```

| Field | Description |
|---|---|
| `protocol_version` | Wire format version; currently `1` |
| `site_id` | Site this node belongs to; `0` = local / unassigned |
| `tcp_port` | Port where this node accepts transport connections |
| `function_flags` | Bitfield of node capabilities (see below) |
| `name_len` | Byte length of the following name string |
| `name` | Node name in `namespace:instance` form, e.g. `v4l2:microscope` |

`function_flags` bits:

| Bit | Mask | Role |
|---|---|---|
| 0 | `0x0001` | Source — produces video |
| 1 | `0x0002` | Relay — receives and distributes streams |
| 2 | `0x0004` | Sink — consumes video |
| 3 | `0x0008` | Controller — has a user-facing control interface |

A node may set multiple bits.

### Multicast Parameters

| Parameter | Value |
|---|---|
| Group | `224.0.0.251` |
| Port | `5353` |
| TTL | `1` (LAN only) |

No Avahi or Bonjour dependency — nodes open a raw UDP multicast socket directly using standard POSIX APIs.

### Site ID Translation

`site_id = 0` means "local / unassigned". Both sides of a site-to-site link will independently default to `0`, so a gateway cannot forward announcements across the boundary without rewriting the field — all nodes on both sides would appear identical.

The gateway assigns a distinct non-zero `site_id` to each side and rewrites `site_id` in all announcements (and any protocol messages carrying a `site_id`) as they cross the boundary. Receiving an announcement with `site_id = 0` on a cross-site link is a gateway protocol error.

---

## Serialisation Primitives

`str8` — a length-prefixed UTF-8 string, not NUL-terminated on the wire:

```mermaid
%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-7: "length"
8-15: "data (≤ 255 bytes)"
```

Maximum string length is 255 bytes.

---

## Control Commands

All requests follow the `CONTROL_REQUEST` frame format (request_id + command + command-specific fields). All responses follow the `CONTROL_RESPONSE` frame format (request_id + status + response-specific fields).

### `ENUM_DEVICES` (0x0003)

**Request** — no extra fields beyond request_id and command.

**Response** on status `OK`:

Repeated `media_count` times (u16):

- `path` str8, `driver` str8, `model` str8, `bus_info` str8
- `vnode_count` u8, then repeated `vnode_count` times:
  - `path` str8, `entity_name` str8
  - `entity_type` u32, `entity_flags` u32, `device_caps` u32
  - `pad_flags` u8, `is_capture` u8

Then repeated `standalone_count` times (u16):

- `path` str8, `name` str8

`device_caps` carries `V4L2_CAP_*` bits from `VIDIOC_QUERYCAP` (using `device_caps` if `V4L2_CAP_DEVICE_CAPS` is set, otherwise `capabilities`). Notable bits: `0x00000001` = `VIDEO_CAPTURE`, `0x00800000` = `META_CAPTURE`.

`is_capture` is `1` if `device_caps & V4L2_CAP_VIDEO_CAPTURE`, else `0`.

### `ENUM_CONTROLS` (0x0004)

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0004"
32-47: "device_index"
```

**Response** on status `OK` — repeated `count` times (u16):

Fixed prefix per control:

```mermaid
%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-31: "id"
32-39: "type"
40-71: "flags"
```

Followed by `name` str8, then the fixed suffix:

```mermaid
packet-beta
0-31: "min"
32-63: "max"
64-95: "step"
96-127: "default_val"
128-159: "current_val"
160-167: "menu_count"
```

Followed by `menu_count` menu items. Each menu item: `index` u32, then `name` str8, then `int_value` i64.

`type` values match `V4L2_CTRL_TYPE_*`:

| Value | Type |
|---|---|
| `1` | `INTEGER` — use a slider |
| `2` | `BOOLEAN` — use a checkbox |
| `3` | `MENU` — use a select; `name` of each menu item is the label |
| `4` | `BUTTON` — use a button |
| `9` | `INTEGER_MENU` — use a select; `int_value` of each menu item is the label |

`flags` bits (from `V4L2_CTRL_FLAG_*`): `0x0001` = disabled, `0x0002` = grabbed (read-only due to another active control), `0x0004` = read-only.

For `MENU` and `INTEGER_MENU` controls, set the control value to a menu item's `index` (not its `int_value`). `int_value` is informational display text only.

Menu items may have non-contiguous `index` values (gaps where the driver returns `EINVAL` for `VIDIOC_QUERYMENU`).

### `GET_CONTROL` (0x0005)

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0005"
32-47: "device_index"
48-79: "control_id"
```

**Response** on status `OK`:

```mermaid
packet-beta
0-15: "request_id"
16-31: "status"
32-63: "value"
```

### `SET_CONTROL` (0x0006)

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0006"
32-47: "device_index"
48-79: "control_id"
80-111: "value"
```

**Response** — no extra fields beyond request_id and status.

For `MENU` and `INTEGER_MENU` controls, `value` must be a valid menu item `index` as returned by `ENUM_CONTROLS`.

### `START_INGEST` (0x0008)

Sets wanted state on a source node: open the specified V4L2 device, configure the stream format, and connect outbound to the given sink.

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0008"
32-47: "stream_id"
48-63: "format"
64-79: "width"
80-95: "height"
96-111: "fps_n"
112-127: "fps_d"
128-143: "dest_port"
144-159: "transport_mode"
160-167: "device_path_len"
168-175: "device_path …"
```

Followed by `dest_host` str8.

| Field | Description |
|---|---|
| `stream_id` | ID assigned by the controller; used in all subsequent `VIDEO_FRAME` messages |
| `format` | Codec format code (see [Codec Formats](#codec-formats)); `0` = auto-select best MJPEG |
| `width` | Capture width in pixels; `0` = auto-select |
| `height` | Capture height in pixels; `0` = auto-select |
| `fps_n` | Frame rate numerator; `0` = auto-select |
| `fps_d` | Frame rate denominator |
| `dest_port` | TCP port of the sink node to connect to |
| `transport_mode` | `0x0001` = encapsulated (framed); `0x0002` = opaque (raw byte stream) |
| `device_path` | str8 — path to the V4L2 device, e.g. `/dev/video0` |
| `dest_host` | str8 — hostname or IP of the sink node |

**Response** — no extra fields beyond request_id and status. `OK` means the wanted state was accepted; the node will reconcile asynchronously.

### `STOP_INGEST` (0x0009)

Sets wanted state: stop the ingest stream and disconnect from the sink.

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0009"
32-47: "stream_id"
```

**Response** — no extra fields beyond request_id and status.

### `START_DISPLAY` (0x000A)

Opens a viewer window on a sink node and routes incoming `VIDEO_FRAME` messages for `stream_id` to it.

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x000A"
32-47: "stream_id"
48-63: "win_x (i16)"
64-79: "win_y (i16)"
80-95: "win_w"
96-111: "win_h"
112-119: "scale"
120-127: "anchor"
128-135: "no_signal_fps"
136-143: "reserved"
```

| Field | Description |
|---|---|
| `stream_id` | Stream to display; must match incoming `VIDEO_FRAME` stream_id |
| `win_x`, `win_y` | Window screen position (signed; for multi-monitor placement) |
| `win_w`, `win_h` | Window size in pixels; `0` = default (1280×720) |
| `scale` | `0`=stretch `1`=fit `2`=fill `3`=1:1 |
| `anchor` | `0`=center `1`=topleft |
| `no_signal_fps` | Frame rate of no-signal animation (0 = default 15 fps) |

**Response** — no extra fields beyond request_id and status. `OK` means the display slot was reserved; the window opens asynchronously on the main thread.

### `STOP_DISPLAY` (0x000B)

Closes the viewer window for the given stream.

**Request**:

```mermaid
%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x000B"
32-47: "stream_id"
```

**Response** — no extra fields beyond request_id and status.