video-setup/docs/protocol.md

Protocol Reference

This document describes the full wire protocol used between nodes. All multi-byte integers are little-endian. Serialization primitives (put_u16, get_u32, etc.) are defined in serial.h.

---

Layers

flowchart TD
    A[TCP connection] --> B[Transport layer<br>frame header + length-prefixed payload]
    B --> C[Message type dispatcher<br>routes payload to handler]
    C --> D1[Video frame handler<br>stream_id + compressed data]
    C --> D2[Control request/response handler<br>request_id + command fields]
    C --> D3[Stream event handler<br>stream_id + event_code]

• TCP — reliable byte stream; provides ordering and delivery but no message boundaries
• Transport layer — adds a 6-byte header to every message, giving a length so any node can skip or forward unknown messages
• Message type dispatcher — reads message_type and routes the payload to the appropriate handler
• Handlers — interpret the payload according to their own schema; the transport layer has no knowledge of handler internals

---

Transport Layer

Frame Format

Every message on the wire is a frame:

%%{init: {'packet': {'bitsPerRow': 48}}}%%
packet-beta
0-15: "message_type"
16-47: "payload_length"
48-95: "payload …"

Total header size: 6 bytes. payload_length is the byte count of the payload only — it does not include the 6-byte header itself.

A node that does not recognise message_type can skip the frame by consuming exactly payload_length bytes and discarding them. This allows relays and future nodes to forward or ignore unknown message types without understanding their structure.

Byte Order

All fields are little-endian. This applies to the header and to all fields within payloads.

Connection Model

Each TCP connection carries a single logical channel between two peers. Multiple streams (video, control, events) are multiplexed on the same connection by message_type and by stream/request identifiers inside payloads. There is no connection-level stream identifier in the header — that information belongs to the payload.

---

Message Types

Value	Name	Description
0x0001	VIDEO_FRAME	One compressed video frame for a stream
0x0002	CONTROL_REQUEST	Request from one node to another
0x0003	CONTROL_RESPONSE	Response to a prior control request
0x0004	STREAM_EVENT	Lifecycle signal for a stream (interrupted, resumed)
0x0010	DISCOVERY_ANNOUNCE	UDP multicast node announcement (see Discovery)

Values not listed are reserved. A node receiving an unknown type must skip the payload (payload_length bytes) and continue reading.

---

Payload Schemas

VIDEO_FRAME (0x0001)

%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "stream_id"
16-31: "frame data …"

stream_id identifies which video stream this frame belongs to. The codec is established at stream open time (see Stream Lifecycle) and does not appear in every frame.

CONTROL_REQUEST (0x0002)

packet-beta
0-15: "request_id"
16-31: "command"
32-63: "command-specific …"

request_id is chosen by the sender and echoed in the matching CONTROL_RESPONSE. It is used to correlate responses to requests when multiple requests are in flight simultaneously.

command values:

Value	Command	Description
0x0001	STREAM_OPEN	Open a new video stream on this connection
0x0002	STREAM_CLOSE	Close a video stream
0x0003	ENUM_DEVICES	List V4L2 devices on the remote node
0x0004	ENUM_CONTROLS	List V4L2 controls for a device
0x0005	GET_CONTROL	Get a V4L2 control value
0x0006	SET_CONTROL	Set a V4L2 control value
0x0007	ENUM_MONITORS	List X11 monitors (XRandR) on the remote node

CONTROL_RESPONSE (0x0003)

packet-beta
0-15: "request_id"
16-31: "status"
32-63: "response-specific …"

request_id matches the originating request. status values:

Value	Meaning
0x0000	OK
0x0001	Error — generic failure
0x0002	Error — unknown command
0x0003	Error — invalid parameters
0x0004	Error — resource not found

STREAM_EVENT (0x0004)

%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-15: "stream_id"
16-23: "event_code"
24-55: "event-specific …"

event_code values:

Value	Name	Meaning
0x01	STREAM_INTERRUPTED	Device lost; frames will stop. Receiver should reset parser state and discard any partial frame.
0x02	STREAM_RESUMED	Device recovered; a clean frame follows.

---

Stream Lifecycle

Before video frames can flow, a stream must be opened. This establishes the codec and pixel format so receivers can decode frames without per-frame metadata.

Opening a Stream (STREAM_OPEN request)

packet-beta
0-15: "request_id"
16-31: "command = 0x0001"
32-47: "stream_id"
48-63: "format"
64-79: "pixel_format"
80-95: "origin"

Field	Description
stream_id	Chosen by the sender; identifies this stream on subsequent VIDEO_FRAME and STREAM_EVENT messages
format	Wire format of the compressed frame data (see Codec Formats)
pixel_format	Pixel layout for raw formats; zero for compressed formats (see Pixel Formats)
origin	How the frames were produced; informational only, does not affect decoding (see Origins)

The receiver responds with CONTROL_RESPONSE. On status = OK the stream is open and VIDEO_FRAME messages with that stream_id may follow immediately.

Closing a Stream (STREAM_CLOSE request)

%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0002"
32-47: "stream_id"

After a close response, no further VIDEO_FRAME messages should be sent for that stream_id.

---

Codec Formats

Carried in the format field of STREAM_OPEN.

Value	Format
0x0001	MJPEG
0x0002	H.264
0x0003	H.265 / HEVC
0x0004	AV1
0x0005	FFV1
0x0006	ProRes
0x0007	QOI
0x0008	Raw pixels (requires pixel_format)
0x0009	Raw pixels + ZSTD (requires pixel_format)

Pixel Formats

Carried in the pixel_format field of STREAM_OPEN. Zero and ignored for compressed formats.

Value	Layout
0x0001	BGRA 8:8:8:8
0x0002	RGBA 8:8:8:8
0x0003	BGR 8:8:8
0x0004	YUV 4:2:0 planar
0x0005	YUV 4:2:2 packed

Origins

Carried in the origin field of STREAM_OPEN. Informational — does not affect decoding.

Value	Origin
0x0001	Device native (camera or capture card encoded it directly)
0x0002	libjpeg-turbo
0x0003	ffmpeg (libavcodec)
0x0004	ffmpeg (subprocess)
0x0005	VA-API direct
0x0006	NVENC direct
0x0007	Software (other)

---

Discovery

Node discovery uses UDP multicast. The wire format is a standard transport frame (same 6-byte header) sent to the multicast group rather than a TCP peer.

Announcement Frame (DISCOVERY_ANNOUNCE, 0x0010)

Sent periodically by every node and immediately on startup.

%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-7: "protocol_version"
8-23: "site_id"
24-39: "tcp_port"
40-55: "function_flags"
56-63: "name_len"
64-95: "name …"

Field	Description
protocol_version	Wire format version; currently 1
site_id	Site this node belongs to; 0 = local / unassigned
tcp_port	Port where this node accepts transport connections
function_flags	Bitfield of node capabilities (see below)
name_len	Byte length of the following name string
name	Node name in namespace:instance form, e.g. v4l2:microscope

function_flags bits:

Bit	Mask	Role
0	0x0001	Source — produces video
1	0x0002	Relay — receives and distributes streams
2	0x0004	Sink — consumes video
3	0x0008	Controller — has a user-facing control interface

A node may set multiple bits.

Multicast Parameters

Parameter	Value
Group	224.0.0.251
Port	5353
TTL	1 (LAN only)

No Avahi or Bonjour dependency — nodes open a raw UDP multicast socket directly using standard POSIX APIs.

Site ID Translation

site_id = 0 means "local / unassigned". Both sides of a site-to-site link will independently default to 0, so a gateway cannot forward announcements across the boundary without rewriting the field — all nodes on both sides would appear identical.

The gateway assigns a distinct non-zero site_id to each side and rewrites site_id in all announcements (and any protocol messages carrying a site_id) as they cross the boundary. Receiving an announcement with site_id = 0 on a cross-site link is a gateway protocol error.

---

Serialisation Primitives

str8 — a length-prefixed UTF-8 string, not NUL-terminated on the wire:

%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-7: "length"
8-15: "data (≤ 255 bytes)"

Maximum string length is 255 bytes.

---

Control Commands

All requests follow the CONTROL_REQUEST frame format (request_id + command + command-specific fields). All responses follow the CONTROL_RESPONSE frame format (request_id + status + response-specific fields).

ENUM_DEVICES (0x0003)

Request — no extra fields beyond request_id and command.

Response on status OK:

Repeated media_count times (u16):

• path str8, driver str8, model str8, bus_info str8
• vnode_count u8, then repeated vnode_count times:
  • path str8, entity_name str8
  • entity_type u32, entity_flags u32, device_caps u32
  • pad_flags u8, is_capture u8

Then repeated standalone_count times (u16):

• path str8, name str8

device_caps carries V4L2_CAP_* bits from VIDIOC_QUERYCAP (using device_caps if V4L2_CAP_DEVICE_CAPS is set, otherwise capabilities). Notable bits: 0x00000001 = VIDEO_CAPTURE, 0x00800000 = META_CAPTURE.

is_capture is 1 if device_caps & V4L2_CAP_VIDEO_CAPTURE, else 0.

ENUM_CONTROLS (0x0004)

Request:

%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0004"
32-47: "device_index"

Response on status OK — repeated count times (u16):

Fixed prefix per control:

%%{init: {'packet': {'bitsPerRow': 8}}}%%
packet-beta
0-31: "id"
32-39: "type"
40-71: "flags"

Followed by name str8, then the fixed suffix:

packet-beta
0-31: "min"
32-63: "max"
64-95: "step"
96-127: "default_val"
128-159: "current_val"
160-167: "menu_count"

Followed by menu_count menu items. Each menu item: index u32, then name str8, then int_value i64.

type values match V4L2_CTRL_TYPE_*:

Value	Type
1	INTEGER — use a slider
2	BOOLEAN — use a checkbox
3	MENU — use a select; name of each menu item is the label
4	BUTTON — use a button
9	INTEGER_MENU — use a select; int_value of each menu item is the label

flags bits (from V4L2_CTRL_FLAG_*): 0x0001 = disabled, 0x0002 = grabbed (read-only due to another active control), 0x0004 = read-only.

For MENU and INTEGER_MENU controls, set the control value to a menu item's index (not its int_value). int_value is informational display text only.

Menu items may have non-contiguous index values (gaps where the driver returns EINVAL for VIDIOC_QUERYMENU).

GET_CONTROL (0x0005)

Request:

%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0005"
32-47: "device_index"
48-79: "control_id"

Response on status OK:

packet-beta
0-15: "request_id"
16-31: "status"
32-63: "value"

SET_CONTROL (0x0006)

Request:

%%{init: {'packet': {'bitsPerRow': 16}}}%%
packet-beta
0-15: "request_id"
16-31: "command = 0x0006"
32-47: "device_index"
48-79: "control_id"
80-111: "value"

Response — no extra fields beyond request_id and status.

For MENU and INTEGER_MENU controls, value must be a valid menu item index as returned by ENUM_CONTROLS.