video-setup/docs/protocol.md

Protocol Reference

This document describes the full wire protocol used between nodes. All multi-byte integers are little-endian. Serialization primitives (put_u16, get_u32, etc.) are defined in serial.h.

---

Layers

flowchart TD
    A[TCP connection] --> B[Transport layer<br>frame header + length-prefixed payload]
    B --> C[Message type dispatcher<br>routes payload to handler]
    C --> D1[Video frame handler<br>stream_id + compressed data]
    C --> D2[Control request/response handler<br>request_id + command fields]
    C --> D3[Stream event handler<br>stream_id + event_code]

• TCP — reliable byte stream; provides ordering and delivery but no message boundaries
• Transport layer — adds a 6-byte header to every message, giving a length so any node can skip or forward unknown messages
• Message type dispatcher — reads message_type and routes the payload to the appropriate handler
• Handlers — interpret the payload according to their own schema; the transport layer has no knowledge of handler internals

---

Transport Layer

Frame Format

Every message on the wire is a frame:

+------------------+------------------------+--------------------+
| message_type: u16 | payload_length: u32   | payload: bytes ... |
+------------------+------------------------+--------------------+
        2 bytes              4 bytes            payload_length bytes

Total header size: 6 bytes.

payload_length is the byte count of the payload only — it does not include the 6-byte header itself.

A node that does not recognise message_type can skip the frame by consuming exactly payload_length bytes and discarding them. This allows relays and future nodes to forward or ignore unknown message types without understanding their structure.

Byte Order

All fields are little-endian. This applies to the header and to all fields within payloads.

Connection Model

Each TCP connection carries a single logical channel between two peers. Multiple streams (video, control, events) are multiplexed on the same connection by message_type and by stream/request identifiers inside payloads. There is no connection-level stream identifier in the header — that information belongs to the payload.

---

Message Types

Value	Name	Description
0x0001	VIDEO_FRAME	One compressed video frame for a stream
0x0002	CONTROL_REQUEST	Request from one node to another
0x0003	CONTROL_RESPONSE	Response to a prior control request
0x0004	STREAM_EVENT	Lifecycle signal for a stream (interrupted, resumed)
0x0010	DISCOVERY_ANNOUNCE	UDP multicast node announcement (see Discovery)

Values not listed are reserved. A node receiving an unknown type must skip the payload (payload_length bytes) and continue reading.

---

Payload Schemas

VIDEO_FRAME (0x0001)

+---------------+----------------------------------------------+
| stream_id: u16 | frame data (compressed, codec per stream_open) |
+---------------+----------------------------------------------+
     2 bytes                  payload_length - 2 bytes

stream_id identifies which video stream this frame belongs to. The codec is established at stream open time (see Stream Lifecycle) and does not appear in every frame.

CONTROL_REQUEST (0x0002)

+------------------+----------------+---------------------------+
| request_id: u16  | command: u16   | command-specific fields   |
+------------------+----------------+---------------------------+
       2 bytes           2 bytes         remaining bytes

request_id is chosen by the sender and echoed in the matching CONTROL_RESPONSE. It is used to correlate responses to requests when multiple requests are in flight simultaneously.

command values:

Value	Command	Description
0x0001	STREAM_OPEN	Open a new video stream on this connection
0x0002	STREAM_CLOSE	Close a video stream
0x0003	ENUM_DEVICES	List V4L2 devices on the remote node
0x0004	ENUM_CONTROLS	List V4L2 controls for a device
0x0005	GET_CONTROL	Get a V4L2 control value
0x0006	SET_CONTROL	Set a V4L2 control value
0x0007	ENUM_MONITORS	List X11 monitors (XRandR) on the remote node

CONTROL_RESPONSE (0x0003)

+------------------+---------------+---------------------------+
| request_id: u16  | status: u16   | response-specific fields  |
+------------------+---------------+---------------------------+
       2 bytes          2 bytes          remaining bytes

request_id matches the originating request. status values:

Value	Meaning
0x0000	OK
0x0001	Error — generic failure
0x0002	Error — unknown command
0x0003	Error — invalid parameters
0x0004	Error — resource not found

STREAM_EVENT (0x0004)

+---------------+------------------+---------------------------+
| stream_id: u16 | event_code: u8  | event-specific fields     |
+---------------+------------------+---------------------------+
     2 bytes           1 byte            remaining bytes

event_code values:

Value	Name	Meaning
0x01	STREAM_INTERRUPTED	Device lost; frames will stop. Receiver should reset parser state and discard any partial frame.
0x02	STREAM_RESUMED	Device recovered; a clean frame follows.

---

Stream Lifecycle

Before video frames can flow, a stream must be opened. This establishes the codec and pixel format so receivers can decode frames without per-frame metadata.

Opening a Stream (STREAM_OPEN request)

The sender issues a CONTROL_REQUEST with command STREAM_OPEN:

+------------------+------------------+-----------------+------------------+-------------------+----------------+
| request_id: u16  | command: u16     | stream_id: u16  | format: u16      | pixel_format: u16 | origin: u16    |
+------------------+------------------+-----------------+------------------+-------------------+----------------+

Field	Description
stream_id	Chosen by the sender; identifies this stream on subsequent VIDEO_FRAME and STREAM_EVENT messages
format	Wire format of the compressed frame data (see Codec Formats)
pixel_format	Pixel layout for raw formats; zero for compressed formats (see Pixel Formats)
origin	How the frames were produced; informational only, does not affect decoding (see Origins)

The receiver responds with CONTROL_RESPONSE. On status = OK the stream is open and VIDEO_FRAME messages with that stream_id may follow immediately.

Closing a Stream (STREAM_CLOSE request)

+------------------+------------------+-----------------+
| request_id: u16  | command: u16     | stream_id: u16  |
+------------------+------------------+-----------------+

After a close response, no further VIDEO_FRAME messages should be sent for that stream_id.

---

Codec Formats

Carried in the format field of STREAM_OPEN.

Value	Format
0x0001	MJPEG
0x0002	H.264
0x0003	H.265 / HEVC
0x0004	AV1
0x0005	FFV1
0x0006	ProRes
0x0007	QOI
0x0008	Raw pixels (requires pixel_format)
0x0009	Raw pixels + ZSTD (requires pixel_format)

Pixel Formats

Carried in the pixel_format field of STREAM_OPEN. Zero and ignored for compressed formats.

Value	Layout
0x0001	BGRA 8:8:8:8
0x0002	RGBA 8:8:8:8
0x0003	BGR 8:8:8
0x0004	YUV 4:2:0 planar
0x0005	YUV 4:2:2 packed

Origins

Carried in the origin field of STREAM_OPEN. Informational — does not affect decoding.

Value	Origin
0x0001	Device native (camera or capture card encoded it directly)
0x0002	libjpeg-turbo
0x0003	ffmpeg (libavcodec)
0x0004	ffmpeg (subprocess)
0x0005	VA-API direct
0x0006	NVENC direct
0x0007	Software (other)

---

Discovery

Node discovery uses UDP multicast. The wire format is a standard transport frame (same 6-byte header) sent to the multicast group rather than a TCP peer.

Announcement Frame (DISCOVERY_ANNOUNCE, 0x0010)

Sent periodically by every node and immediately on startup.

+----------------------+------------+-----------------+------------------+-------------------+------------------+
| protocol_version: u8 | site_id: u16 | tcp_port: u16 | function_flags: u16 | name_len: u8  | name: bytes      |
+----------------------+------------+-----------------+------------------+-------------------+------------------+

Field	Description
protocol_version	Wire format version; currently 1
site_id	Site this node belongs to; 0 = local / unassigned
tcp_port	Port where this node accepts transport connections
function_flags	Bitfield of node capabilities (see below)
name_len	Byte length of the following name string
name	Node name in namespace:instance form, e.g. v4l2:microscope

function_flags bits:

Bit	Mask	Role
0	0x0001	Source — produces video
1	0x0002	Relay — receives and distributes streams
2	0x0004	Sink — consumes video
3	0x0008	Controller — has a user-facing control interface

A node may set multiple bits.

Multicast Parameters

Parameter	Value
Group	224.0.0.251
Port	5353
TTL	1 (LAN only)

No Avahi or Bonjour dependency — nodes open a raw UDP multicast socket directly using standard POSIX APIs.

Site ID Translation

site_id = 0 means "local / unassigned". Both sides of a site-to-site link will independently default to 0, so a gateway cannot forward announcements across the boundary without rewriting the field — all nodes on both sides would appear identical.

The gateway assigns a distinct non-zero site_id to each side and rewrites site_id in all announcements (and any protocol messages carrying a site_id) as they cross the boundary. Receiving an announcement with site_id = 0 on a cross-site link is a gateway protocol error.

---

Serialisation Primitives

str8 — a length-prefixed UTF-8 string, not NUL-terminated on the wire:

+------------+-------------------+
| length: u8 | bytes: length × u8 |
+------------+-------------------+

Maximum string length is 255 bytes.

---

Control Commands

All requests follow the CONTROL_REQUEST frame format (request_id + command + command-specific fields). All responses follow the CONTROL_RESPONSE frame format (request_id + status + response-specific fields).

ENUM_DEVICES (0x0003)

Request — no extra fields beyond request_id and command.

Response on status OK:

+-------------------+
| media_count: u16  |
+-------------------+
  × media_count:
    +------------+------------+-----------+-------------+-----------------+
    | path: str8 | driver:str8| model:str8| bus_info:str8| vnode_count: u8 |
    +------------+------------+-----------+-------------+-----------------+
      × vnode_count:
        +------------+-----------------+------------------+-------------------+------------------+----------------+------------------+
        | path: str8 | entity_name:str8| entity_type: u32 | entity_flags: u32 | device_caps: u32 | pad_flags: u8  | is_capture: u8   |
        +------------+-----------------+------------------+-------------------+------------------+----------------+------------------+
+----------------------+
| standalone_count: u16 |
+----------------------+
  × standalone_count:
    +------------+------------+
    | path: str8 | name: str8 |
    +------------+------------+

device_caps carries V4L2_CAP_* bits from VIDIOC_QUERYCAP (using device_caps if V4L2_CAP_DEVICE_CAPS is set, otherwise capabilities). Notable bits: 0x00000001 = VIDEO_CAPTURE, 0x00800000 = META_CAPTURE.

is_capture is 1 if device_caps & V4L2_CAP_VIDEO_CAPTURE, else 0.

ENUM_CONTROLS (0x0004)

Request:

+--------------------+
| device_index: u16  |
+--------------------+

Response on status OK:

+---------------+
| count: u16    |
+---------------+
  × count:
    +---------+---------+----------+------------+----------+---------+----------+-------------+-----------------+------------------+
    | id: u32 | type: u8 | flags: u32 | name: str8 | min: i32 | max: i32 | step: i32 | default: i32 | current: i32 | menu_count: u8 |
    +---------+---------+----------+------------+----------+---------+----------+-------------+-----------------+------------------+
      × menu_count:
        +------------+------------+------------------+
        | index: u32 | name: str8 | int_value: i64   |
        +------------+------------+------------------+

type values match V4L2_CTRL_TYPE_*:

Value	Type
1	INTEGER — use a slider
2	BOOLEAN — use a checkbox
3	MENU — use a select; name of each menu item is the label
4	BUTTON — use a button
9	INTEGER_MENU — use a select; int_value of each menu item is the label

flags bits (from V4L2_CTRL_FLAG_*): 0x0001 = disabled, 0x0002 = grabbed (read-only due to another active control), 0x0004 = read-only.

For MENU and INTEGER_MENU controls, set the control value to a menu item's index (not its int_value). int_value is informational display text only.

Menu items may have non-contiguous index values (gaps where the driver returns EINVAL for VIDIOC_QUERYMENU).

GET_CONTROL (0x0005)

Request:

+--------------------+------------------+
| device_index: u16  | control_id: u32  |
+--------------------+------------------+

Response on status OK:

+-------------+
| value: i32  |
+-------------+

SET_CONTROL (0x0006)

Request:

+--------------------+------------------+-------------+
| device_index: u16  | control_id: u32  | value: i32  |
+--------------------+------------------+-------------+

Response — no extra fields beyond request_id and status.

For MENU and INTEGER_MENU controls, value must be a valid menu item index as returned by ENUM_CONTROLS.