mute
7fa50dd4e4
OPEN and OPEN_ACK now carry proto:1 alongside upstream_id:1. The session is tagged with its transport proto and reliability semantics switch on the specific proto: - TCP (proto=1): full L2 reliability (seq, ack, retransmit, in-order) - UDP (proto=2, reserved): best-effort (no retransmit, no ordering, no pure ACKs — seq/ack_seq fields present but ignored) This is architecturally correct: instead of a generic 'reliable:bool' flag, each proto gets the semantics it needs. Today only TCP exists so every session is reliable, but the extension point is clean for when stateless protos are added. Updated: PROTOCOL.md, Rust frame.rs/session.rs/main.rs, C# Frame.cs/SessionManager.cs.
370 lines
15 KiB
Markdown
370 lines
15 KiB
Markdown
# gatuna wire protocol
|
||
|
||
All frames ride Ethernet with ethertype `0x6969`. The ethertype is the sole
|
||
discriminator; there is no magic number inside the payload.
|
||
|
||
## Frame layout (non-DATA frames)
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+---------------+-------------------------------+
|
||
| version | type | session_id |
|
||
+---------------+---------------+-------------------------------+
|
||
| payload_len (big-endian) | payload ... |
|
||
+-------------------------------+ +
|
||
| |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **version** (u8): protocol version. Currently `2`.
|
||
- **type** (u8): frame type, see table below.
|
||
- **session_id** (u32, big-endian): `0` for non-session frames; the
|
||
server-assigned ID for session-scoped frames.
|
||
- **payload_len** (u16, big-endian): number of payload bytes that follow.
|
||
The receiver reads exactly this many bytes and ignores any trailing
|
||
Ethernet padding (frames under 60 bytes are zero-padded by the NIC to
|
||
meet the minimum Ethernet frame size).
|
||
- **payload** (`payload_len` bytes): type-dependent.
|
||
|
||
## DATA frame layout
|
||
|
||
DATA frames carry two additional fields after the common header for L2-level
|
||
reliability:
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+---------------+-------------------------------+
|
||
| version=2 | type=0x06 | session_id |
|
||
+---------------+---------------+-------------------------------+
|
||
| payload_len (big-endian) | seq (big-endian) |
|
||
+-------------------------------+-------------------------------+
|
||
| ack_seq (big-endian) | payload ... |
|
||
+-------------------------------+ +
|
||
| |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **seq** (u32, big-endian): monotonically increasing per-session sequence
|
||
number. Wraps at 2^32 (same as TCP). Identifies this DATA frame's position
|
||
in the byte stream.
|
||
- **ack_seq** (u32, big-endian): cumulative acknowledgment — the highest
|
||
contiguous `seq` that the sender of this frame has delivered to its local
|
||
TCP socket. The receiver uses this to advance its retransmit window.
|
||
- **payload** (`payload_len` bytes): raw application bytes (0–1480). A
|
||
`payload_len = 0` DATA frame is a **pure ACK** — it carries no data, just
|
||
an acknowledgment. This mirrors TCP's empty-segment ACK.
|
||
|
||
Maximum payload: 1500 (Ethernet MTU) − 8 (common header) − 8 (seq + ack_seq)
|
||
= **1484 bytes**. In practice we cap at **1480** to stay conservative.
|
||
|
||
## Frame types
|
||
|
||
| Type | Name | Direction | session_id | Payload |
|
||
|------|-------------|---------------|------------|----------------------------------|
|
||
| 0x01 | DISCOVER | C → broadcast | 0 | empty |
|
||
| 0x02 | MANIFEST | S → C | 0 | `hostname_len:1, hostname:N, entries...` |
|
||
| 0x03 | OPEN | C → S | 0 | `upstream_id:1, proto:1` |
|
||
| 0x04 | OPEN_ACK | S → C | assigned | `upstream_id:1, proto:1` |
|
||
| 0x05 | OPEN_NAK | S → C | 0 | `upstream_id:1, reason:1` |
|
||
| 0x06 | DATA | both | session | `seq:4, ack_seq:4, raw bytes` |
|
||
| 0x07 | CLOSE | both | session | optional `reason:1` |
|
||
| 0x0B | PING | C → S | 0 | `nonce:8` |
|
||
| 0x0C | PONG | S → C | 0 | `nonce:8` (echoed) |
|
||
|
||
Reserved (unimplemented; parse returns Err, encode unimplemented):
|
||
|
||
| Type | Name |
|
||
|------|-------------|
|
||
| 0x08 | UDP_OPEN |
|
||
| 0x09 | UDP_DATA |
|
||
| 0x0A | UDP_CLOSE |
|
||
|
||
## L2 reliability
|
||
|
||
The reliability semantics of a session depend on its transport proto,
|
||
which is carried end-to-end through OPEN/OPEN_ACK:
|
||
|
||
- **TCP (`proto = 1`):** DATA frames are delivered reliably and in-order via
|
||
a TCP-like sequence + cumulative ACK + retransmit mechanism. This ensures
|
||
that a dropped Ethernet frame does not permanently corrupt a TCP session
|
||
(which would otherwise happen because the local kernel TCP stack ACKs data
|
||
before it is chunked into DATA frames).
|
||
- **UDP (`proto = 2`, reserved):** DATA frames are delivered best-effort. The
|
||
`seq` and `ack_seq` fields are present in the frame layout for uniformity
|
||
but are ignored — no retransmit, no in-order buffering, no pure ACKs. Drops
|
||
are tolerated because stateless protocols either don't care or handle
|
||
recovery at the application layer.
|
||
|
||
The following sections describe the TCP reliability mechanism.
|
||
|
||
### Sender state (per session)
|
||
|
||
- `send_seq`: next seq to assign (starts at 0, increments per DATA frame).
|
||
- `retransmit_buffer`: map of `seq → (payload, timestamp)`, holding all sent
|
||
but unacked frames.
|
||
- `acked_seq`: highest seq acknowledged by the peer (initially `None`).
|
||
|
||
On sending DATA:
|
||
1. Assign `seq = send_seq; send_seq += 1`.
|
||
2. Set `ack_seq` to the highest contiguous seq we have received from the peer
|
||
(our receive side's `deliver_seq`).
|
||
3. Store `(payload, now)` in `retransmit_buffer[seq]`.
|
||
4. Transmit the frame.
|
||
|
||
On receiving an `ack_seq` in any DATA frame (including pure ACKs):
|
||
1. Advance `acked_seq` to `max(acked_seq, ack_seq)`.
|
||
2. Remove all entries from `retransmit_buffer` with `seq <= ack_seq`.
|
||
|
||
Retransmit timer (per session, checked periodically):
|
||
1. For each entry in `retransmit_buffer` older than `RETRANSMIT_TIMEOUT`
|
||
(default 5 ms), retransmit the frame and reset its timestamp.
|
||
2. If any entry has been retransmitted more than `MAX_RETRIES` times (default
|
||
10), send `CLOSE` and tear down the session.
|
||
|
||
### Receiver state (per session)
|
||
|
||
- `expected_seq`: next seq expected (starts at 0).
|
||
- `receive_buffer`: map of `seq → payload`, holding out-of-order frames.
|
||
- `deliver_seq`: highest seq delivered to the local TCP socket (starts at
|
||
`None`; reported as `ack_seq` in outgoing DATA frames).
|
||
|
||
On receiving DATA with `seq`:
|
||
1. If `seq < expected_seq`: duplicate (already delivered). Discard the payload,
|
||
but still process the `ack_seq` field to advance the send window. Send a
|
||
pure ACK so the sender can converge.
|
||
2. If `seq == expected_seq`: deliver payload to the TCP socket. Increment
|
||
`expected_seq`. Then check `receive_buffer` for the next contiguous seq and
|
||
deliver those too (drain the buffer in order). Update `deliver_seq`.
|
||
3. If `seq > expected_seq`: store in `receive_buffer[seq]`. Do not deliver yet.
|
||
Send a pure ACK (re-ACKing `deliver_seq`) to trigger retransmit of the gap.
|
||
|
||
### Pure ACK frames
|
||
|
||
When a side needs to ACK but has no data to send, it sends a DATA frame with
|
||
`payload_len = 0`. The `seq` field is set to `send_seq` (consuming a seq
|
||
number, same as TCP's empty segment) and `ack_seq` carries the cumulative
|
||
acknowledgment. The receiver processes the `ack_seq` and discards the empty
|
||
payload without delivering to the TCP socket.
|
||
|
||
### Acknowledgment timing
|
||
|
||
- When data is flowing in both directions, each DATA frame carries the latest
|
||
`ack_seq` — no separate ACK frames needed.
|
||
- When data is one-directional, the receiver sends a pure ACK after each DATA
|
||
frame (or after a small batch, implementation-defined).
|
||
- On receiving a duplicate or out-of-order frame, the receiver immediately
|
||
sends a pure ACK to help the sender converge.
|
||
|
||
### Why this is not the Two Generals Problem
|
||
|
||
We do not need mutual consensus. We need one-sided reliable delivery: the
|
||
sender retransmits until it gets an ACK. If the ACK is lost, the sender
|
||
retransmits the data; the receiver sees a duplicate, discards it, and re-ACKs.
|
||
This converges in O(1) round trips. The only unsolvable case — the last frame
|
||
before a permanent link death — is handled by `MAX_RETRIES` → `CLOSE`, which is
|
||
correct: a dead link should kill the session.
|
||
|
||
## Payload field encodings
|
||
|
||
### MANIFEST payload
|
||
|
||
A hostname prefix followed by variable-length entries, parsed sequentially
|
||
until the payload is consumed.
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+-----------------------------------------------+
|
||
| hostname_len | hostname (UTF-8, hostname_len bytes) ... |
|
||
+---------------+-----------------------------------------------+
|
||
| id | proto | port (big-endian) |
|
||
+---------------+---------------+-------------------------------+
|
||
| label_len | label (UTF-8, label_len bytes) ... |
|
||
+---------------+-----------------------------------------------+
|
||
| ... repeated ... |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **hostname_len** (u8): length in bytes of the server's hostname. `0` is valid
|
||
(unknown hostname).
|
||
- **hostname** (`hostname_len` bytes, UTF-8): the server's hostname, read via
|
||
`gethostname(2)` at startup. Maximum 255 bytes.
|
||
- **id** (u8): upstream identifier (1-based positional index from `gatunad`
|
||
cmdline).
|
||
- **proto** (u8): `1 = TCP`, `2 = UDP` (reserved; not emitted in v1).
|
||
- **port** (u16, big-endian): the real port on the server's `127.0.0.1`.
|
||
- **label_len** (u8): length in bytes of the label that follows. `0` means no
|
||
label.
|
||
- **label** (`label_len` bytes, UTF-8): human-readable name for the upstream,
|
||
taken from the `PORT[:label]` cmdline argument. Maximum 255 bytes.
|
||
|
||
To parse: read `hostname_len`, then `hostname_len` bytes of hostname, then read
|
||
5-byte fixed entry prefixes + `label_len` bytes of label each, repeating until
|
||
the payload is exhausted. The number of entries is not carried explicitly.
|
||
|
||
### OPEN payload
|
||
|
||
```
|
||
+---------------+---------------+
|
||
| upstream_id | proto |
|
||
+---------------+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): which MANIFEST entry to open.
|
||
- **proto** (u8): the transport protocol of the upstream (`1 = TCP`,
|
||
`2 = UDP`). Carried end-to-end so both sides know which reliability
|
||
semantics apply to the session. Must match the proto advertised in the
|
||
MANIFEST for that upstream.
|
||
|
||
### OPEN_ACK payload
|
||
|
||
```
|
||
+---------------+---------------+
|
||
| upstream_id | proto |
|
||
+---------------+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): echoes the requested upstream.
|
||
- **proto** (u8): echoes the requested proto. The session is identified by
|
||
the `session_id` field in the header, not the payload.
|
||
|
||
### OPEN_NAK payload
|
||
|
||
```
|
||
+---------------+---------------+
|
||
| upstream_id | reason |
|
||
+---------------+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): echoes the requested upstream.
|
||
- **reason** (u8): see reason codes.
|
||
|
||
### DATA payload
|
||
|
||
`[ seq:4 ][ ack_seq:4 ][ raw bytes ]` — the `seq` and `ack_seq` fields are
|
||
part of the DATA frame's extended header (between `payload_len` and the
|
||
payload). The `payload_len` field counts only the raw bytes, not the seq/ack
|
||
fields. Up to 1480 bytes of application data per frame.
|
||
|
||
### CLOSE payload
|
||
|
||
```
|
||
+---------------+
|
||
| reason? |
|
||
+---------------+
|
||
```
|
||
|
||
- **reason** (u8, optional): present iff payload length ≥ 1. See reason codes.
|
||
|
||
### PING / PONG payload
|
||
|
||
```
|
||
+ +
|
||
| nonce (big-endian, 8 bytes) |
|
||
+ +
|
||
```
|
||
|
||
- **nonce** (u64, big-endian): arbitrary value chosen by the client. The
|
||
server echoes it verbatim in the PONG reply. Used to correlate RTT
|
||
measurements.
|
||
|
||
## Reason codes
|
||
|
||
| Value | Meaning |
|
||
|-------|-------------------|
|
||
| 0 | unspecified |
|
||
| 1 | unknown_upstream |
|
||
| 2 | connect_failed |
|
||
| 3 | oversize |
|
||
| 4 | unknown_session |
|
||
| 5 | max_retries |
|
||
|
||
## Discovery flow
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| DISCOVER (dst = broadcast) |
|
||
|-------------------------------->|
|
||
| |
|
||
| MANIFEST (unicast) |
|
||
|<--------------------------------|
|
||
| |
|
||
```
|
||
|
||
The server learns the client's MAC from the DISCOVER frame's source address and
|
||
unicasts the MANIFEST back. The server never speaks unsolicited.
|
||
|
||
## TCP session lifecycle
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| OPEN { upstream_id } |
|
||
|-------------------------------->|
|
||
| | TcpStream::connect(127.0.0.1:port)
|
||
| |
|
||
| OPEN_ACK { session_id } |
|
||
|<--------------------------------| (on success)
|
||
| OR |
|
||
| OPEN_NAK { reason } |
|
||
|<--------------------------------| (on failure)
|
||
| |
|
||
| DATA { seq, ack_seq, bytes } |
|
||
|<------------------------------->| DATA { seq, ack_seq, bytes }
|
||
| |
|
||
| CLOSE { session_id, reason? } |
|
||
|<------------------------------->| (on EOF, RST, or max_retries)
|
||
| |
|
||
```
|
||
|
||
- `session_id` is allocated by the server as a monotonically increasing u32
|
||
(starting at 1) from an atomic counter. Collision by wraparound is ignored.
|
||
- `seq` starts at 0 on both sides of each session and increments per DATA
|
||
frame (including pure ACKs).
|
||
- Either side may send `CLOSE`. The side receiving `CLOSE` tears down its half
|
||
and stops emitting frames for that session.
|
||
- The server's socket→tunnel pump reads `TcpStream` in 1480-byte chunks and
|
||
emits one DATA frame per chunk. On `read` returning 0 (FIN) or an error,
|
||
it emits `CLOSE` and exits.
|
||
- The server's tunnel→socket path delivers DATA payloads in seq order to the
|
||
`TcpStream` with `write_all`. On error it emits `CLOSE` and drops the session.
|
||
- A DATA frame is retransmitted if no ACK is received within 5 ms. After 10
|
||
failed retransmits, the session is closed with `reason = max_retries`.
|
||
|
||
## Network test (PING/PONG)
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| PING { nonce } |
|
||
|-------------------------------->|
|
||
| |
|
||
| PONG { nonce } |
|
||
|<--------------------------------|
|
||
| |
|
||
| (repeated at random intervals) |
|
||
| |
|
||
```
|
||
|
||
- The client sends `PING` frames at random 10–100 ms intervals, each with a
|
||
unique `nonce`.
|
||
- The server echoes the nonce verbatim in a `PONG` frame.
|
||
- The client correlates `PONG` nonces with outstanding `PING` timestamps to
|
||
compute RTT, average latency, jitter (mean absolute delta of consecutive
|
||
RTTs), and drop rate (unanswered PINGs).
|
||
- PING/PONG frames use `session_id = 0`; they are independent of TCP sessions.
|
||
|
||
## Version compatibility
|
||
|
||
- Version 1: no `seq`/`ack_seq` in DATA frames, no L2 reliability. Dropped
|
||
DATA frames kill the TCP session.
|
||
- Version 2: DATA frames carry `seq` + `ack_seq`, L2 retransmit. Dropped
|
||
frames are recovered.
|
||
|
||
Version 2 is the current version. A receiver that sees `version != 2` rejects
|
||
the frame. Both sides of a session must speak the same version; there is no
|
||
negotiation.
|