gatuna/PROTOCOL.md

gatuna wire protocol

All frames ride Ethernet with ethertype 0x6969. The ethertype is the sole discriminator; there is no magic number inside the payload.

Frame layout (non-DATA frames)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------------------------------+
| version       | type          | session_id                    |
+---------------+---------------+-------------------------------+
| payload_len (big-endian)      | payload ...                   |
+-------------------------------+                               +
|                                                               |
+---------------------------------------------------------------+

• version (u8): protocol version. Currently 2.
• type (u8): frame type, see table below.
• session_id (u32, big-endian): 0 for non-session frames; the server-assigned ID for session-scoped frames.
• payload_len (u16, big-endian): number of payload bytes that follow. The receiver reads exactly this many bytes and ignores any trailing Ethernet padding (frames under 60 bytes are zero-padded by the NIC to meet the minimum Ethernet frame size).
• payload (payload_len bytes): type-dependent.

DATA frame layout

DATA frames carry two additional fields after the common header for L2-level reliability:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------------------------------+
| version=2     | type=0x06     | session_id                    |
+---------------+---------------+-------------------------------+
| payload_len (big-endian)      | seq (big-endian)              |
+-------------------------------+-------------------------------+
| ack_seq (big-endian)          | payload ...                   |
+-------------------------------+                               +
|                                                               |
+---------------------------------------------------------------+

• seq (u32, big-endian): monotonically increasing per-session sequence number. Wraps at 2^32 (same as TCP). Identifies this DATA frame's position in the byte stream.
• ack_seq (u32, big-endian): cumulative acknowledgment — the highest contiguous seq that the sender of this frame has delivered to its local TCP socket. The receiver uses this to advance its retransmit window.
• payload (payload_len bytes): raw application bytes (0–1480). A payload_len = 0 DATA frame is a pure ACK — it carries no data, just an acknowledgment. This mirrors TCP's empty-segment ACK.

Maximum payload: 1500 (Ethernet MTU) − 8 (common header) − 8 (seq + ack_seq) = 1484 bytes. In practice we cap at 1480 to stay conservative.

Frame types

Type	Name	Direction	session_id	Payload
0x01	DISCOVER	C → broadcast	0	empty
0x02	MANIFEST	S → C	0	hostname_len:1, hostname:N, entries...
0x03	OPEN	C → S	0	upstream_id:1, proto:1
0x04	OPEN_ACK	S → C	assigned	upstream_id:1, proto:1
0x05	OPEN_NAK	S → C	0	upstream_id:1, reason:1
0x06	DATA	both	session	seq:4, ack_seq:4, raw bytes
0x07	CLOSE	both	session	optional reason:1
0x0B	PING	C → S	0	nonce:8
0x0C	PONG	S → C	0	nonce:8 (echoed)

Reserved (unimplemented; parse returns Err, encode unimplemented):

Type	Name
0x08	UDP_OPEN
0x09	UDP_DATA
0x0A	UDP_CLOSE

L2 reliability

The reliability semantics of a session depend on its transport proto, which is carried end-to-end through OPEN/OPEN_ACK:

• TCP (proto = 1): DATA frames are delivered reliably and in-order via a TCP-like sequence + cumulative ACK + retransmit mechanism. This ensures that a dropped Ethernet frame does not permanently corrupt a TCP session (which would otherwise happen because the local kernel TCP stack ACKs data before it is chunked into DATA frames).
• UDP (proto = 2, reserved): DATA frames are delivered best-effort. The seq and ack_seq fields are present in the frame layout for uniformity but are ignored — no retransmit, no in-order buffering, no pure ACKs. Drops are tolerated because stateless protocols either don't care or handle recovery at the application layer.

The following sections describe the TCP reliability mechanism.

Sender state (per session)

• send_seq: next seq to assign (starts at 0, increments per DATA frame).
• retransmit_buffer: map of seq → (payload, timestamp), holding all sent but unacked frames.
• acked_seq: highest seq acknowledged by the peer (initially None).

On sending DATA:

1. Assign seq = send_seq; send_seq += 1.
2. Set ack_seq to the highest contiguous seq we have received from the peer (our receive side's deliver_seq).
3. Store (payload, now) in retransmit_buffer[seq].
4. Transmit the frame.

On receiving an ack_seq in any DATA frame (including pure ACKs):

1. Advance acked_seq to max(acked_seq, ack_seq).
2. Remove all entries from retransmit_buffer with seq <= ack_seq.

Retransmit timer (per session, checked periodically):

1. For each entry in retransmit_buffer older than RETRANSMIT_TIMEOUT (default 5 ms), retransmit the frame and reset its timestamp.
2. If any entry has been retransmitted more than MAX_RETRIES times (default 10), send CLOSE and tear down the session.

Receiver state (per session)

• expected_seq: next seq expected (starts at 0).
• receive_buffer: map of seq → payload, holding out-of-order frames.
• deliver_seq: highest seq delivered to the local TCP socket (starts at None; reported as ack_seq in outgoing DATA frames).

On receiving DATA with seq:

1. If seq < expected_seq: duplicate (already delivered). Discard the payload, but still process the ack_seq field to advance the send window. Send a pure ACK so the sender can converge.
2. If seq == expected_seq: deliver payload to the TCP socket. Increment expected_seq. Then check receive_buffer for the next contiguous seq and deliver those too (drain the buffer in order). Update deliver_seq.
3. If seq > expected_seq: store in receive_buffer[seq]. Do not deliver yet. Send a pure ACK (re-ACKing deliver_seq) to trigger retransmit of the gap.

Pure ACK frames

When a side needs to ACK but has no data to send, it sends a DATA frame with payload_len = 0. The seq field is set to send_seq (consuming a seq number, same as TCP's empty segment) and ack_seq carries the cumulative acknowledgment. The receiver processes the ack_seq and discards the empty payload without delivering to the TCP socket.

Acknowledgment timing

• When data is flowing in both directions, each DATA frame carries the latest ack_seq — no separate ACK frames needed.
• When data is one-directional, the receiver sends a pure ACK after each DATA frame (or after a small batch, implementation-defined).
• On receiving a duplicate or out-of-order frame, the receiver immediately sends a pure ACK to help the sender converge.

Why this is not the Two Generals Problem

We do not need mutual consensus. We need one-sided reliable delivery: the sender retransmits until it gets an ACK. If the ACK is lost, the sender retransmits the data; the receiver sees a duplicate, discards it, and re-ACKs. This converges in O(1) round trips. The only unsolvable case — the last frame before a permanent link death — is handled by MAX_RETRIES → CLOSE, which is correct: a dead link should kill the session.

Payload field encodings

MANIFEST payload

A hostname prefix followed by variable-length entries, parsed sequentially until the payload is consumed.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+-----------------------------------------------+
| hostname_len  | hostname (UTF-8, hostname_len bytes) ...       |
+---------------+-----------------------------------------------+
| id            | proto         | port (big-endian)             |
+---------------+---------------+-------------------------------+
| label_len     | label (UTF-8, label_len bytes) ...           |
+---------------+-----------------------------------------------+
| ... repeated ...                                             |
+---------------------------------------------------------------+

• hostname_len (u8): length in bytes of the server's hostname. 0 is valid (unknown hostname).
• hostname (hostname_len bytes, UTF-8): the server's hostname, read via gethostname(2) at startup. Maximum 255 bytes.
• id (u8): upstream identifier (1-based positional index from gatunad cmdline).
• proto (u8): 1 = TCP, 2 = UDP (reserved; not emitted in v1).
• port (u16, big-endian): the real port on the server's 127.0.0.1.
• label_len (u8): length in bytes of the label that follows. 0 means no label.
• label (label_len bytes, UTF-8): human-readable name for the upstream, taken from the PORT[:label] cmdline argument. Maximum 255 bytes.

To parse: read hostname_len, then hostname_len bytes of hostname, then read 5-byte fixed entry prefixes + label_len bytes of label each, repeating until the payload is exhausted. The number of entries is not carried explicitly.

OPEN payload

+---------------+---------------+
| upstream_id   | proto         |
+---------------+---------------+

• upstream_id (u8): which MANIFEST entry to open.
• proto (u8): the transport protocol of the upstream (1 = TCP, 2 = UDP). Carried end-to-end so both sides know which reliability semantics apply to the session. Must match the proto advertised in the MANIFEST for that upstream.

OPEN_ACK payload

+---------------+---------------+
| upstream_id   | proto         |
+---------------+---------------+

• upstream_id (u8): echoes the requested upstream.
• proto (u8): echoes the requested proto. The session is identified by the session_id field in the header, not the payload.

OPEN_NAK payload

+---------------+---------------+
| upstream_id   | reason        |
+---------------+---------------+

• upstream_id (u8): echoes the requested upstream.
• reason (u8): see reason codes.

DATA payload

[ seq:4 ][ ack_seq:4 ][ raw bytes ] — the seq and ack_seq fields are part of the DATA frame's extended header (between payload_len and the payload). The payload_len field counts only the raw bytes, not the seq/ack fields. Up to 1480 bytes of application data per frame.

CLOSE payload

+---------------+
| reason?       |
+---------------+

• reason (u8, optional): present iff payload length ≥ 1. See reason codes.

PING / PONG payload

+                                                               +
| nonce (big-endian, 8 bytes)                                   |
+                                                               +

• nonce (u64, big-endian): arbitrary value chosen by the client. The server echoes it verbatim in the PONG reply. Used to correlate RTT measurements.

Reason codes

Value	Meaning
0	unspecified
1	unknown_upstream
2	connect_failed
3	oversize
4	unknown_session
5	max_retries

Discovery flow

client                            server
  |                                 |
  | DISCOVER (dst = broadcast)      |
  |-------------------------------->|
  |                                 |
  |        MANIFEST (unicast)       |
  |<--------------------------------|
  |                                 |

The server learns the client's MAC from the DISCOVER frame's source address and unicasts the MANIFEST back. The server never speaks unsolicited.

TCP session lifecycle

client                            server
  |                                 |
  | OPEN { upstream_id }            |
  |-------------------------------->|
  |                                 | TcpStream::connect(127.0.0.1:port)
  |                                 |
  |   OPEN_ACK { session_id }       |
  |<--------------------------------|   (on success)
  |        OR                       |
  |   OPEN_NAK { reason }           |
  |<--------------------------------|   (on failure)
  |                                 |
  | DATA { seq, ack_seq, bytes }    |
  |<------------------------------->| DATA { seq, ack_seq, bytes }
  |                                 |
  | CLOSE { session_id, reason? }   |
  |<------------------------------->|   (on EOF, RST, or max_retries)
  |                                 |

• session_id is allocated by the server as a monotonically increasing u32 (starting at 1) from an atomic counter. Collision by wraparound is ignored.
• seq starts at 0 on both sides of each session and increments per DATA frame (including pure ACKs).
• Either side may send CLOSE. The side receiving CLOSE tears down its half and stops emitting frames for that session.
• The server's socket→tunnel pump reads TcpStream in 1480-byte chunks and emits one DATA frame per chunk. On read returning 0 (FIN) or an error, it emits CLOSE and exits.
• The server's tunnel→socket path delivers DATA payloads in seq order to the TcpStream with write_all. On error it emits CLOSE and drops the session.
• A DATA frame is retransmitted if no ACK is received within 5 ms. After 10 failed retransmits, the session is closed with reason = max_retries.

Network test (PING/PONG)

client                            server
  |                                 |
  | PING { nonce }                  |
  |-------------------------------->|
  |                                 |
  |        PONG { nonce }           |
  |<--------------------------------|
  |                                 |
  | (repeated at random intervals)  |
  |                                 |

• The client sends PING frames at random 10–100 ms intervals, each with a unique nonce.
• The server echoes the nonce verbatim in a PONG frame.
• The client correlates PONG nonces with outstanding PING timestamps to compute RTT, average latency, jitter (mean absolute delta of consecutive RTTs), and drop rate (unanswered PINGs).
• PING/PONG frames use session_id = 0; they are independent of TCP sessions.

Version compatibility

• Version 1: no seq/ack_seq in DATA frames, no L2 reliability. Dropped DATA frames kill the TCP session.
• Version 2: DATA frames carry seq + ack_seq, L2 retransmit. Dropped frames are recovered.

Version 2 is the current version. A receiver that sees version != 2 rejects the frame. Both sides of a session must speak the same version; there is no negotiation.