mute
eb8994d1e1
DATA frames now carry seq:4 and ack_seq:4 in a 16-byte extended header. Both sides maintain per-session send/recv state: Sender: - Monotonic seq counter, retransmit buffer (seq -> frame bytes) - Retransmit timer: 5ms timeout, 10 max retries -> CLOSE - Window advances on cumulative ACK Receiver: - In-order delivery to TCP socket (expected_seq) - Out-of-order buffering (SortedList by seq) - Duplicate detection (seq < expected -> discard + re-ACK) - Pure ACK frames (empty-payload DATA) for duplicate/OOO responses This prevents lost Ethernet frames from permanently corrupting TCP sessions, which was the key v1 limitation. The local kernel TCP stack ACKs data before we chunk it into DATA frames; without L2 reliability a dropped frame creates an unrecoverable gap. Version bumped to 2. Both sides must speak v2; no negotiation. Updated: PROTOCOL.md (full v2 spec), README.md, Rust frame.rs/ session.rs/main.rs, C# Frame.cs/SessionManager.cs/TunnelLink.cs.
355 lines
14 KiB
Markdown
355 lines
14 KiB
Markdown
# gatuna wire protocol
|
||
|
||
All frames ride Ethernet with ethertype `0x6969`. The ethertype is the sole
|
||
discriminator; there is no magic number inside the payload.
|
||
|
||
## Frame layout (non-DATA frames)
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+---------------+-------------------------------+
|
||
| version | type | session_id |
|
||
+---------------+---------------+-------------------------------+
|
||
| payload_len (big-endian) | payload ... |
|
||
+-------------------------------+ +
|
||
| |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **version** (u8): protocol version. Currently `2`.
|
||
- **type** (u8): frame type, see table below.
|
||
- **session_id** (u32, big-endian): `0` for non-session frames; the
|
||
server-assigned ID for session-scoped frames.
|
||
- **payload_len** (u16, big-endian): number of payload bytes that follow.
|
||
The receiver reads exactly this many bytes and ignores any trailing
|
||
Ethernet padding (frames under 60 bytes are zero-padded by the NIC to
|
||
meet the minimum Ethernet frame size).
|
||
- **payload** (`payload_len` bytes): type-dependent.
|
||
|
||
## DATA frame layout
|
||
|
||
DATA frames carry two additional fields after the common header for L2-level
|
||
reliability:
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+---------------+-------------------------------+
|
||
| version=2 | type=0x06 | session_id |
|
||
+---------------+---------------+-------------------------------+
|
||
| payload_len (big-endian) | seq (big-endian) |
|
||
+-------------------------------+-------------------------------+
|
||
| ack_seq (big-endian) | payload ... |
|
||
+-------------------------------+ +
|
||
| |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **seq** (u32, big-endian): monotonically increasing per-session sequence
|
||
number. Wraps at 2^32 (same as TCP). Identifies this DATA frame's position
|
||
in the byte stream.
|
||
- **ack_seq** (u32, big-endian): cumulative acknowledgment — the highest
|
||
contiguous `seq` that the sender of this frame has delivered to its local
|
||
TCP socket. The receiver uses this to advance its retransmit window.
|
||
- **payload** (`payload_len` bytes): raw application bytes (0–1480). A
|
||
`payload_len = 0` DATA frame is a **pure ACK** — it carries no data, just
|
||
an acknowledgment. This mirrors TCP's empty-segment ACK.
|
||
|
||
Maximum payload: 1500 (Ethernet MTU) − 8 (common header) − 8 (seq + ack_seq)
|
||
= **1484 bytes**. In practice we cap at **1480** to stay conservative.
|
||
|
||
## Frame types
|
||
|
||
| Type | Name | Direction | session_id | Payload |
|
||
|------|-------------|---------------|------------|----------------------------------|
|
||
| 0x01 | DISCOVER | C → broadcast | 0 | empty |
|
||
| 0x02 | MANIFEST | S → C | 0 | `hostname_len:1, hostname:N, entries...` |
|
||
| 0x03 | OPEN | C → S | 0 | `upstream_id:1` |
|
||
| 0x04 | OPEN_ACK | S → C | assigned | `upstream_id:1` |
|
||
| 0x05 | OPEN_NAK | S → C | 0 | `upstream_id:1, reason:1` |
|
||
| 0x06 | DATA | both | session | `seq:4, ack_seq:4, raw bytes` |
|
||
| 0x07 | CLOSE | both | session | optional `reason:1` |
|
||
| 0x0B | PING | C → S | 0 | `nonce:8` |
|
||
| 0x0C | PONG | S → C | 0 | `nonce:8` (echoed) |
|
||
|
||
Reserved (unimplemented; parse returns Err, encode unimplemented):
|
||
|
||
| Type | Name |
|
||
|------|-------------|
|
||
| 0x08 | UDP_OPEN |
|
||
| 0x09 | UDP_DATA |
|
||
| 0x0A | UDP_CLOSE |
|
||
|
||
## L2 reliability
|
||
|
||
DATA frames are delivered reliably and in-order via a TCP-like sequence +
|
||
cumulative ACK + retransmit mechanism. This ensures that a dropped Ethernet
|
||
frame does not permanently corrupt a TCP session (which would otherwise happen
|
||
because the local kernel TCP stack ACKs data before it is chunked into DATA
|
||
frames).
|
||
|
||
### Sender state (per session)
|
||
|
||
- `send_seq`: next seq to assign (starts at 0, increments per DATA frame).
|
||
- `retransmit_buffer`: map of `seq → (payload, timestamp)`, holding all sent
|
||
but unacked frames.
|
||
- `acked_seq`: highest seq acknowledged by the peer (initially `None`).
|
||
|
||
On sending DATA:
|
||
1. Assign `seq = send_seq; send_seq += 1`.
|
||
2. Set `ack_seq` to the highest contiguous seq we have received from the peer
|
||
(our receive side's `deliver_seq`).
|
||
3. Store `(payload, now)` in `retransmit_buffer[seq]`.
|
||
4. Transmit the frame.
|
||
|
||
On receiving an `ack_seq` in any DATA frame (including pure ACKs):
|
||
1. Advance `acked_seq` to `max(acked_seq, ack_seq)`.
|
||
2. Remove all entries from `retransmit_buffer` with `seq <= ack_seq`.
|
||
|
||
Retransmit timer (per session, checked periodically):
|
||
1. For each entry in `retransmit_buffer` older than `RETRANSMIT_TIMEOUT`
|
||
(default 5 ms), retransmit the frame and reset its timestamp.
|
||
2. If any entry has been retransmitted more than `MAX_RETRIES` times (default
|
||
10), send `CLOSE` and tear down the session.
|
||
|
||
### Receiver state (per session)
|
||
|
||
- `expected_seq`: next seq expected (starts at 0).
|
||
- `receive_buffer`: map of `seq → payload`, holding out-of-order frames.
|
||
- `deliver_seq`: highest seq delivered to the local TCP socket (starts at
|
||
`None`; reported as `ack_seq` in outgoing DATA frames).
|
||
|
||
On receiving DATA with `seq`:
|
||
1. If `seq < expected_seq`: duplicate (already delivered). Discard the payload,
|
||
but still process the `ack_seq` field to advance the send window. Send a
|
||
pure ACK so the sender can converge.
|
||
2. If `seq == expected_seq`: deliver payload to the TCP socket. Increment
|
||
`expected_seq`. Then check `receive_buffer` for the next contiguous seq and
|
||
deliver those too (drain the buffer in order). Update `deliver_seq`.
|
||
3. If `seq > expected_seq`: store in `receive_buffer[seq]`. Do not deliver yet.
|
||
Send a pure ACK (re-ACKing `deliver_seq`) to trigger retransmit of the gap.
|
||
|
||
### Pure ACK frames
|
||
|
||
When a side needs to ACK but has no data to send, it sends a DATA frame with
|
||
`payload_len = 0`. The `seq` field is set to `send_seq` (consuming a seq
|
||
number, same as TCP's empty segment) and `ack_seq` carries the cumulative
|
||
acknowledgment. The receiver processes the `ack_seq` and discards the empty
|
||
payload without delivering to the TCP socket.
|
||
|
||
### Acknowledgment timing
|
||
|
||
- When data is flowing in both directions, each DATA frame carries the latest
|
||
`ack_seq` — no separate ACK frames needed.
|
||
- When data is one-directional, the receiver sends a pure ACK after each DATA
|
||
frame (or after a small batch, implementation-defined).
|
||
- On receiving a duplicate or out-of-order frame, the receiver immediately
|
||
sends a pure ACK to help the sender converge.
|
||
|
||
### Why this is not the Two Generals Problem
|
||
|
||
We do not need mutual consensus. We need one-sided reliable delivery: the
|
||
sender retransmits until it gets an ACK. If the ACK is lost, the sender
|
||
retransmits the data; the receiver sees a duplicate, discards it, and re-ACKs.
|
||
This converges in O(1) round trips. The only unsolvable case — the last frame
|
||
before a permanent link death — is handled by `MAX_RETRIES` → `CLOSE`, which is
|
||
correct: a dead link should kill the session.
|
||
|
||
## Payload field encodings
|
||
|
||
### MANIFEST payload
|
||
|
||
A hostname prefix followed by variable-length entries, parsed sequentially
|
||
until the payload is consumed.
|
||
|
||
```
|
||
0 1 2 3
|
||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||
+---------------+-----------------------------------------------+
|
||
| hostname_len | hostname (UTF-8, hostname_len bytes) ... |
|
||
+---------------+-----------------------------------------------+
|
||
| id | proto | port (big-endian) |
|
||
+---------------+---------------+-------------------------------+
|
||
| label_len | label (UTF-8, label_len bytes) ... |
|
||
+---------------+-----------------------------------------------+
|
||
| ... repeated ... |
|
||
+---------------------------------------------------------------+
|
||
```
|
||
|
||
- **hostname_len** (u8): length in bytes of the server's hostname. `0` is valid
|
||
(unknown hostname).
|
||
- **hostname** (`hostname_len` bytes, UTF-8): the server's hostname, read via
|
||
`gethostname(2)` at startup. Maximum 255 bytes.
|
||
- **id** (u8): upstream identifier (1-based positional index from `gatunad`
|
||
cmdline).
|
||
- **proto** (u8): `1 = TCP`, `2 = UDP` (reserved; not emitted in v1).
|
||
- **port** (u16, big-endian): the real port on the server's `127.0.0.1`.
|
||
- **label_len** (u8): length in bytes of the label that follows. `0` means no
|
||
label.
|
||
- **label** (`label_len` bytes, UTF-8): human-readable name for the upstream,
|
||
taken from the `PORT[:label]` cmdline argument. Maximum 255 bytes.
|
||
|
||
To parse: read `hostname_len`, then `hostname_len` bytes of hostname, then read
|
||
5-byte fixed entry prefixes + `label_len` bytes of label each, repeating until
|
||
the payload is exhausted. The number of entries is not carried explicitly.
|
||
|
||
### OPEN payload
|
||
|
||
```
|
||
+---------------+
|
||
| upstream_id |
|
||
+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): which MANIFEST entry to open.
|
||
|
||
### OPEN_ACK payload
|
||
|
||
```
|
||
+---------------+
|
||
| upstream_id |
|
||
+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): echoes the requested upstream. The session is
|
||
identified by the `session_id` field in the header, not the payload.
|
||
|
||
### OPEN_NAK payload
|
||
|
||
```
|
||
+---------------+---------------+
|
||
| upstream_id | reason |
|
||
+---------------+---------------+
|
||
```
|
||
|
||
- **upstream_id** (u8): echoes the requested upstream.
|
||
- **reason** (u8): see reason codes.
|
||
|
||
### DATA payload
|
||
|
||
`[ seq:4 ][ ack_seq:4 ][ raw bytes ]` — the `seq` and `ack_seq` fields are
|
||
part of the DATA frame's extended header (between `payload_len` and the
|
||
payload). The `payload_len` field counts only the raw bytes, not the seq/ack
|
||
fields. Up to 1480 bytes of application data per frame.
|
||
|
||
### CLOSE payload
|
||
|
||
```
|
||
+---------------+
|
||
| reason? |
|
||
+---------------+
|
||
```
|
||
|
||
- **reason** (u8, optional): present iff payload length ≥ 1. See reason codes.
|
||
|
||
### PING / PONG payload
|
||
|
||
```
|
||
+ +
|
||
| nonce (big-endian, 8 bytes) |
|
||
+ +
|
||
```
|
||
|
||
- **nonce** (u64, big-endian): arbitrary value chosen by the client. The
|
||
server echoes it verbatim in the PONG reply. Used to correlate RTT
|
||
measurements.
|
||
|
||
## Reason codes
|
||
|
||
| Value | Meaning |
|
||
|-------|-------------------|
|
||
| 0 | unspecified |
|
||
| 1 | unknown_upstream |
|
||
| 2 | connect_failed |
|
||
| 3 | oversize |
|
||
| 4 | unknown_session |
|
||
| 5 | max_retries |
|
||
|
||
## Discovery flow
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| DISCOVER (dst = broadcast) |
|
||
|-------------------------------->|
|
||
| |
|
||
| MANIFEST (unicast) |
|
||
|<--------------------------------|
|
||
| |
|
||
```
|
||
|
||
The server learns the client's MAC from the DISCOVER frame's source address and
|
||
unicasts the MANIFEST back. The server never speaks unsolicited.
|
||
|
||
## TCP session lifecycle
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| OPEN { upstream_id } |
|
||
|-------------------------------->|
|
||
| | TcpStream::connect(127.0.0.1:port)
|
||
| |
|
||
| OPEN_ACK { session_id } |
|
||
|<--------------------------------| (on success)
|
||
| OR |
|
||
| OPEN_NAK { reason } |
|
||
|<--------------------------------| (on failure)
|
||
| |
|
||
| DATA { seq, ack_seq, bytes } |
|
||
|<------------------------------->| DATA { seq, ack_seq, bytes }
|
||
| |
|
||
| CLOSE { session_id, reason? } |
|
||
|<------------------------------->| (on EOF, RST, or max_retries)
|
||
| |
|
||
```
|
||
|
||
- `session_id` is allocated by the server as a monotonically increasing u32
|
||
(starting at 1) from an atomic counter. Collision by wraparound is ignored.
|
||
- `seq` starts at 0 on both sides of each session and increments per DATA
|
||
frame (including pure ACKs).
|
||
- Either side may send `CLOSE`. The side receiving `CLOSE` tears down its half
|
||
and stops emitting frames for that session.
|
||
- The server's socket→tunnel pump reads `TcpStream` in 1480-byte chunks and
|
||
emits one DATA frame per chunk. On `read` returning 0 (FIN) or an error,
|
||
it emits `CLOSE` and exits.
|
||
- The server's tunnel→socket path delivers DATA payloads in seq order to the
|
||
`TcpStream` with `write_all`. On error it emits `CLOSE` and drops the session.
|
||
- A DATA frame is retransmitted if no ACK is received within 5 ms. After 10
|
||
failed retransmits, the session is closed with `reason = max_retries`.
|
||
|
||
## Network test (PING/PONG)
|
||
|
||
```
|
||
client server
|
||
| |
|
||
| PING { nonce } |
|
||
|-------------------------------->|
|
||
| |
|
||
| PONG { nonce } |
|
||
|<--------------------------------|
|
||
| |
|
||
| (repeated at random intervals) |
|
||
| |
|
||
```
|
||
|
||
- The client sends `PING` frames at random 10–100 ms intervals, each with a
|
||
unique `nonce`.
|
||
- The server echoes the nonce verbatim in a `PONG` frame.
|
||
- The client correlates `PONG` nonces with outstanding `PING` timestamps to
|
||
compute RTT, average latency, jitter (mean absolute delta of consecutive
|
||
RTTs), and drop rate (unanswered PINGs).
|
||
- PING/PONG frames use `session_id = 0`; they are independent of TCP sessions.
|
||
|
||
## Version compatibility
|
||
|
||
- Version 1: no `seq`/`ack_seq` in DATA frames, no L2 reliability. Dropped
|
||
DATA frames kill the TCP session.
|
||
- Version 2: DATA frames carry `seq` + `ack_seq`, L2 retransmit. Dropped
|
||
frames are recovered.
|
||
|
||
Version 2 is the current version. A receiver that sees `version != 2` rejects
|
||
the frame. Both sides of a session must speak the same version; there is no
|
||
negotiation.
|