# gatuna wire protocol All frames ride Ethernet with ethertype `0x6969`. The ethertype is the sole discriminator; there is no magic number inside the payload. ## Frame layout (non-DATA frames) ``` 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+-------------------------------+ | version | type | session_id | +---------------+---------------+-------------------------------+ | payload_len (big-endian) | payload ... | +-------------------------------+ + | | +---------------------------------------------------------------+ ``` - **version** (u8): protocol version. Currently `2`. - **type** (u8): frame type, see table below. - **session_id** (u32, big-endian): `0` for non-session frames; the server-assigned ID for session-scoped frames. - **payload_len** (u16, big-endian): number of payload bytes that follow. The receiver reads exactly this many bytes and ignores any trailing Ethernet padding (frames under 60 bytes are zero-padded by the NIC to meet the minimum Ethernet frame size). - **payload** (`payload_len` bytes): type-dependent. ## DATA frame layout DATA frames carry two additional fields after the common header for L2-level reliability: ``` 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+-------------------------------+ | version=2 | type=0x06 | session_id | +---------------+---------------+-------------------------------+ | payload_len (big-endian) | seq (big-endian) | +-------------------------------+-------------------------------+ | ack_seq (big-endian) | payload ... | +-------------------------------+ + | | +---------------------------------------------------------------+ ``` - **seq** (u32, big-endian): monotonically increasing per-session sequence number. Wraps at 2^32 (same as TCP). Identifies this DATA frame's position in the byte stream. - **ack_seq** (u32, big-endian): cumulative acknowledgment — the highest contiguous `seq` that the sender of this frame has delivered to its local TCP socket. The receiver uses this to advance its retransmit window. - **payload** (`payload_len` bytes): raw application bytes (0–1480). A `payload_len = 0` DATA frame is a **pure ACK** — it carries no data, just an acknowledgment. This mirrors TCP's empty-segment ACK. Maximum payload: 1500 (Ethernet MTU) − 8 (common header) − 8 (seq + ack_seq) = **1484 bytes**. In practice we cap at **1480** to stay conservative. ## Frame types | Type | Name | Direction | session_id | Payload | |------|-------------|---------------|------------|----------------------------------| | 0x01 | DISCOVER | C → broadcast | 0 | empty | | 0x02 | MANIFEST | S → C | 0 | `hostname_len:1, hostname:N, entries...` | | 0x03 | OPEN | C → S | 0 | `upstream_id:1` | | 0x04 | OPEN_ACK | S → C | assigned | `upstream_id:1` | | 0x05 | OPEN_NAK | S → C | 0 | `upstream_id:1, reason:1` | | 0x06 | DATA | both | session | `seq:4, ack_seq:4, raw bytes` | | 0x07 | CLOSE | both | session | optional `reason:1` | | 0x0B | PING | C → S | 0 | `nonce:8` | | 0x0C | PONG | S → C | 0 | `nonce:8` (echoed) | Reserved (unimplemented; parse returns Err, encode unimplemented): | Type | Name | |------|-------------| | 0x08 | UDP_OPEN | | 0x09 | UDP_DATA | | 0x0A | UDP_CLOSE | ## L2 reliability DATA frames are delivered reliably and in-order via a TCP-like sequence + cumulative ACK + retransmit mechanism. This ensures that a dropped Ethernet frame does not permanently corrupt a TCP session (which would otherwise happen because the local kernel TCP stack ACKs data before it is chunked into DATA frames). ### Sender state (per session) - `send_seq`: next seq to assign (starts at 0, increments per DATA frame). - `retransmit_buffer`: map of `seq → (payload, timestamp)`, holding all sent but unacked frames. - `acked_seq`: highest seq acknowledged by the peer (initially `None`). On sending DATA: 1. Assign `seq = send_seq; send_seq += 1`. 2. Set `ack_seq` to the highest contiguous seq we have received from the peer (our receive side's `deliver_seq`). 3. Store `(payload, now)` in `retransmit_buffer[seq]`. 4. Transmit the frame. On receiving an `ack_seq` in any DATA frame (including pure ACKs): 1. Advance `acked_seq` to `max(acked_seq, ack_seq)`. 2. Remove all entries from `retransmit_buffer` with `seq <= ack_seq`. Retransmit timer (per session, checked periodically): 1. For each entry in `retransmit_buffer` older than `RETRANSMIT_TIMEOUT` (default 5 ms), retransmit the frame and reset its timestamp. 2. If any entry has been retransmitted more than `MAX_RETRIES` times (default 10), send `CLOSE` and tear down the session. ### Receiver state (per session) - `expected_seq`: next seq expected (starts at 0). - `receive_buffer`: map of `seq → payload`, holding out-of-order frames. - `deliver_seq`: highest seq delivered to the local TCP socket (starts at `None`; reported as `ack_seq` in outgoing DATA frames). On receiving DATA with `seq`: 1. If `seq < expected_seq`: duplicate (already delivered). Discard the payload, but still process the `ack_seq` field to advance the send window. Send a pure ACK so the sender can converge. 2. If `seq == expected_seq`: deliver payload to the TCP socket. Increment `expected_seq`. Then check `receive_buffer` for the next contiguous seq and deliver those too (drain the buffer in order). Update `deliver_seq`. 3. If `seq > expected_seq`: store in `receive_buffer[seq]`. Do not deliver yet. Send a pure ACK (re-ACKing `deliver_seq`) to trigger retransmit of the gap. ### Pure ACK frames When a side needs to ACK but has no data to send, it sends a DATA frame with `payload_len = 0`. The `seq` field is set to `send_seq` (consuming a seq number, same as TCP's empty segment) and `ack_seq` carries the cumulative acknowledgment. The receiver processes the `ack_seq` and discards the empty payload without delivering to the TCP socket. ### Acknowledgment timing - When data is flowing in both directions, each DATA frame carries the latest `ack_seq` — no separate ACK frames needed. - When data is one-directional, the receiver sends a pure ACK after each DATA frame (or after a small batch, implementation-defined). - On receiving a duplicate or out-of-order frame, the receiver immediately sends a pure ACK to help the sender converge. ### Why this is not the Two Generals Problem We do not need mutual consensus. We need one-sided reliable delivery: the sender retransmits until it gets an ACK. If the ACK is lost, the sender retransmits the data; the receiver sees a duplicate, discards it, and re-ACKs. This converges in O(1) round trips. The only unsolvable case — the last frame before a permanent link death — is handled by `MAX_RETRIES` → `CLOSE`, which is correct: a dead link should kill the session. ## Payload field encodings ### MANIFEST payload A hostname prefix followed by variable-length entries, parsed sequentially until the payload is consumed. ``` 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+-----------------------------------------------+ | hostname_len | hostname (UTF-8, hostname_len bytes) ... | +---------------+-----------------------------------------------+ | id | proto | port (big-endian) | +---------------+---------------+-------------------------------+ | label_len | label (UTF-8, label_len bytes) ... | +---------------+-----------------------------------------------+ | ... repeated ... | +---------------------------------------------------------------+ ``` - **hostname_len** (u8): length in bytes of the server's hostname. `0` is valid (unknown hostname). - **hostname** (`hostname_len` bytes, UTF-8): the server's hostname, read via `gethostname(2)` at startup. Maximum 255 bytes. - **id** (u8): upstream identifier (1-based positional index from `gatunad` cmdline). - **proto** (u8): `1 = TCP`, `2 = UDP` (reserved; not emitted in v1). - **port** (u16, big-endian): the real port on the server's `127.0.0.1`. - **label_len** (u8): length in bytes of the label that follows. `0` means no label. - **label** (`label_len` bytes, UTF-8): human-readable name for the upstream, taken from the `PORT[:label]` cmdline argument. Maximum 255 bytes. To parse: read `hostname_len`, then `hostname_len` bytes of hostname, then read 5-byte fixed entry prefixes + `label_len` bytes of label each, repeating until the payload is exhausted. The number of entries is not carried explicitly. ### OPEN payload ``` +---------------+ | upstream_id | +---------------+ ``` - **upstream_id** (u8): which MANIFEST entry to open. ### OPEN_ACK payload ``` +---------------+ | upstream_id | +---------------+ ``` - **upstream_id** (u8): echoes the requested upstream. The session is identified by the `session_id` field in the header, not the payload. ### OPEN_NAK payload ``` +---------------+---------------+ | upstream_id | reason | +---------------+---------------+ ``` - **upstream_id** (u8): echoes the requested upstream. - **reason** (u8): see reason codes. ### DATA payload `[ seq:4 ][ ack_seq:4 ][ raw bytes ]` — the `seq` and `ack_seq` fields are part of the DATA frame's extended header (between `payload_len` and the payload). The `payload_len` field counts only the raw bytes, not the seq/ack fields. Up to 1480 bytes of application data per frame. ### CLOSE payload ``` +---------------+ | reason? | +---------------+ ``` - **reason** (u8, optional): present iff payload length ≥ 1. See reason codes. ### PING / PONG payload ``` + + | nonce (big-endian, 8 bytes) | + + ``` - **nonce** (u64, big-endian): arbitrary value chosen by the client. The server echoes it verbatim in the PONG reply. Used to correlate RTT measurements. ## Reason codes | Value | Meaning | |-------|-------------------| | 0 | unspecified | | 1 | unknown_upstream | | 2 | connect_failed | | 3 | oversize | | 4 | unknown_session | | 5 | max_retries | ## Discovery flow ``` client server | | | DISCOVER (dst = broadcast) | |-------------------------------->| | | | MANIFEST (unicast) | |<--------------------------------| | | ``` The server learns the client's MAC from the DISCOVER frame's source address and unicasts the MANIFEST back. The server never speaks unsolicited. ## TCP session lifecycle ``` client server | | | OPEN { upstream_id } | |-------------------------------->| | | TcpStream::connect(127.0.0.1:port) | | | OPEN_ACK { session_id } | |<--------------------------------| (on success) | OR | | OPEN_NAK { reason } | |<--------------------------------| (on failure) | | | DATA { seq, ack_seq, bytes } | |<------------------------------->| DATA { seq, ack_seq, bytes } | | | CLOSE { session_id, reason? } | |<------------------------------->| (on EOF, RST, or max_retries) | | ``` - `session_id` is allocated by the server as a monotonically increasing u32 (starting at 1) from an atomic counter. Collision by wraparound is ignored. - `seq` starts at 0 on both sides of each session and increments per DATA frame (including pure ACKs). - Either side may send `CLOSE`. The side receiving `CLOSE` tears down its half and stops emitting frames for that session. - The server's socket→tunnel pump reads `TcpStream` in 1480-byte chunks and emits one DATA frame per chunk. On `read` returning 0 (FIN) or an error, it emits `CLOSE` and exits. - The server's tunnel→socket path delivers DATA payloads in seq order to the `TcpStream` with `write_all`. On error it emits `CLOSE` and drops the session. - A DATA frame is retransmitted if no ACK is received within 5 ms. After 10 failed retransmits, the session is closed with `reason = max_retries`. ## Network test (PING/PONG) ``` client server | | | PING { nonce } | |-------------------------------->| | | | PONG { nonce } | |<--------------------------------| | | | (repeated at random intervals) | | | ``` - The client sends `PING` frames at random 10–100 ms intervals, each with a unique `nonce`. - The server echoes the nonce verbatim in a `PONG` frame. - The client correlates `PONG` nonces with outstanding `PING` timestamps to compute RTT, average latency, jitter (mean absolute delta of consecutive RTTs), and drop rate (unanswered PINGs). - PING/PONG frames use `session_id = 0`; they are independent of TCP sessions. ## Version compatibility - Version 1: no `seq`/`ack_seq` in DATA frames, no L2 reliability. Dropped DATA frames kill the TCP session. - Version 2: DATA frames carry `seq` + `ack_seq`, L2 retransmit. Dropped frames are recovered. Version 2 is the current version. A receiver that sees `version != 2` rejects the frame. Both sides of a session must speak the same version; there is no negotiation.