add L2 reliability: seq + cumulative ACK + retransmit (protocol v2)
DATA frames now carry seq:4 and ack_seq:4 in a 16-byte extended header. Both sides maintain per-session send/recv state: Sender: - Monotonic seq counter, retransmit buffer (seq -> frame bytes) - Retransmit timer: 5ms timeout, 10 max retries -> CLOSE - Window advances on cumulative ACK Receiver: - In-order delivery to TCP socket (expected_seq) - Out-of-order buffering (SortedList by seq) - Duplicate detection (seq < expected -> discard + re-ACK) - Pure ACK frames (empty-payload DATA) for duplicate/OOO responses This prevents lost Ethernet frames from permanently corrupting TCP sessions, which was the key v1 limitation. The local kernel TCP stack ACKs data before we chunk it into DATA frames; without L2 reliability a dropped frame creates an unrecoverable gap. Version bumped to 2. Both sides must speak v2; no negotiation. Updated: PROTOCOL.md (full v2 spec), README.md, Rust frame.rs/ session.rs/main.rs, C# Frame.cs/SessionManager.cs/TunnelLink.cs.
This commit is contained in:
+135
-18
@@ -3,7 +3,7 @@
|
||||
All frames ride Ethernet with ethertype `0x6969`. The ethertype is the sole
|
||||
discriminator; there is no magic number inside the payload.
|
||||
|
||||
## Frame layout
|
||||
## Frame layout (non-DATA frames)
|
||||
|
||||
```
|
||||
0 1 2 3
|
||||
@@ -17,7 +17,7 @@ discriminator; there is no magic number inside the payload.
|
||||
+---------------------------------------------------------------+
|
||||
```
|
||||
|
||||
- **version** (u8): protocol version. Currently `1`.
|
||||
- **version** (u8): protocol version. Currently `2`.
|
||||
- **type** (u8): frame type, see table below.
|
||||
- **session_id** (u32, big-endian): `0` for non-session frames; the
|
||||
server-assigned ID for session-scoped frames.
|
||||
@@ -27,13 +27,37 @@ discriminator; there is no magic number inside the payload.
|
||||
meet the minimum Ethernet frame size).
|
||||
- **payload** (`payload_len` bytes): type-dependent.
|
||||
|
||||
No CRC, no retransmit, no ordering at this layer. TCP reliability is handled by
|
||||
the endpoints' TCP stacks; UDP (reserved) will rely on application-level
|
||||
mechanisms.
|
||||
## DATA frame layout
|
||||
|
||||
Maximum payload: 1500 (Ethernet MTU) − 8 (our header) = **1492 bytes**. In
|
||||
practice we cap at **1480** to stay conservative. Larger payloads are not
|
||||
emitted in v1.
|
||||
DATA frames carry two additional fields after the common header for L2-level
|
||||
reliability:
|
||||
|
||||
```
|
||||
0 1 2 3
|
||||
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
||||
+---------------+---------------+-------------------------------+
|
||||
| version=2 | type=0x06 | session_id |
|
||||
+---------------+---------------+-------------------------------+
|
||||
| payload_len (big-endian) | seq (big-endian) |
|
||||
+-------------------------------+-------------------------------+
|
||||
| ack_seq (big-endian) | payload ... |
|
||||
+-------------------------------+ +
|
||||
| |
|
||||
+---------------------------------------------------------------+
|
||||
```
|
||||
|
||||
- **seq** (u32, big-endian): monotonically increasing per-session sequence
|
||||
number. Wraps at 2^32 (same as TCP). Identifies this DATA frame's position
|
||||
in the byte stream.
|
||||
- **ack_seq** (u32, big-endian): cumulative acknowledgment — the highest
|
||||
contiguous `seq` that the sender of this frame has delivered to its local
|
||||
TCP socket. The receiver uses this to advance its retransmit window.
|
||||
- **payload** (`payload_len` bytes): raw application bytes (0–1480). A
|
||||
`payload_len = 0` DATA frame is a **pure ACK** — it carries no data, just
|
||||
an acknowledgment. This mirrors TCP's empty-segment ACK.
|
||||
|
||||
Maximum payload: 1500 (Ethernet MTU) − 8 (common header) − 8 (seq + ack_seq)
|
||||
= **1484 bytes**. In practice we cap at **1480** to stay conservative.
|
||||
|
||||
## Frame types
|
||||
|
||||
@@ -44,12 +68,12 @@ emitted in v1.
|
||||
| 0x03 | OPEN | C → S | 0 | `upstream_id:1` |
|
||||
| 0x04 | OPEN_ACK | S → C | assigned | `upstream_id:1` |
|
||||
| 0x05 | OPEN_NAK | S → C | 0 | `upstream_id:1, reason:1` |
|
||||
| 0x06 | DATA | both | session | raw bytes (≤1480) |
|
||||
| 0x06 | DATA | both | session | `seq:4, ack_seq:4, raw bytes` |
|
||||
| 0x07 | CLOSE | both | session | optional `reason:1` |
|
||||
| 0x0B | PING | C → S | 0 | `nonce:8` |
|
||||
| 0x0C | PONG | S → C | 0 | `nonce:8` (echoed) |
|
||||
|
||||
Reserved (unimplemented in v1; parse returns Err, encode unimplemented):
|
||||
Reserved (unimplemented; parse returns Err, encode unimplemented):
|
||||
|
||||
| Type | Name |
|
||||
|------|-------------|
|
||||
@@ -57,6 +81,81 @@ Reserved (unimplemented in v1; parse returns Err, encode unimplemented):
|
||||
| 0x09 | UDP_DATA |
|
||||
| 0x0A | UDP_CLOSE |
|
||||
|
||||
## L2 reliability
|
||||
|
||||
DATA frames are delivered reliably and in-order via a TCP-like sequence +
|
||||
cumulative ACK + retransmit mechanism. This ensures that a dropped Ethernet
|
||||
frame does not permanently corrupt a TCP session (which would otherwise happen
|
||||
because the local kernel TCP stack ACKs data before it is chunked into DATA
|
||||
frames).
|
||||
|
||||
### Sender state (per session)
|
||||
|
||||
- `send_seq`: next seq to assign (starts at 0, increments per DATA frame).
|
||||
- `retransmit_buffer`: map of `seq → (payload, timestamp)`, holding all sent
|
||||
but unacked frames.
|
||||
- `acked_seq`: highest seq acknowledged by the peer (initially `None`).
|
||||
|
||||
On sending DATA:
|
||||
1. Assign `seq = send_seq; send_seq += 1`.
|
||||
2. Set `ack_seq` to the highest contiguous seq we have received from the peer
|
||||
(our receive side's `deliver_seq`).
|
||||
3. Store `(payload, now)` in `retransmit_buffer[seq]`.
|
||||
4. Transmit the frame.
|
||||
|
||||
On receiving an `ack_seq` in any DATA frame (including pure ACKs):
|
||||
1. Advance `acked_seq` to `max(acked_seq, ack_seq)`.
|
||||
2. Remove all entries from `retransmit_buffer` with `seq <= ack_seq`.
|
||||
|
||||
Retransmit timer (per session, checked periodically):
|
||||
1. For each entry in `retransmit_buffer` older than `RETRANSMIT_TIMEOUT`
|
||||
(default 5 ms), retransmit the frame and reset its timestamp.
|
||||
2. If any entry has been retransmitted more than `MAX_RETRIES` times (default
|
||||
10), send `CLOSE` and tear down the session.
|
||||
|
||||
### Receiver state (per session)
|
||||
|
||||
- `expected_seq`: next seq expected (starts at 0).
|
||||
- `receive_buffer`: map of `seq → payload`, holding out-of-order frames.
|
||||
- `deliver_seq`: highest seq delivered to the local TCP socket (starts at
|
||||
`None`; reported as `ack_seq` in outgoing DATA frames).
|
||||
|
||||
On receiving DATA with `seq`:
|
||||
1. If `seq < expected_seq`: duplicate (already delivered). Discard the payload,
|
||||
but still process the `ack_seq` field to advance the send window. Send a
|
||||
pure ACK so the sender can converge.
|
||||
2. If `seq == expected_seq`: deliver payload to the TCP socket. Increment
|
||||
`expected_seq`. Then check `receive_buffer` for the next contiguous seq and
|
||||
deliver those too (drain the buffer in order). Update `deliver_seq`.
|
||||
3. If `seq > expected_seq`: store in `receive_buffer[seq]`. Do not deliver yet.
|
||||
Send a pure ACK (re-ACKing `deliver_seq`) to trigger retransmit of the gap.
|
||||
|
||||
### Pure ACK frames
|
||||
|
||||
When a side needs to ACK but has no data to send, it sends a DATA frame with
|
||||
`payload_len = 0`. The `seq` field is set to `send_seq` (consuming a seq
|
||||
number, same as TCP's empty segment) and `ack_seq` carries the cumulative
|
||||
acknowledgment. The receiver processes the `ack_seq` and discards the empty
|
||||
payload without delivering to the TCP socket.
|
||||
|
||||
### Acknowledgment timing
|
||||
|
||||
- When data is flowing in both directions, each DATA frame carries the latest
|
||||
`ack_seq` — no separate ACK frames needed.
|
||||
- When data is one-directional, the receiver sends a pure ACK after each DATA
|
||||
frame (or after a small batch, implementation-defined).
|
||||
- On receiving a duplicate or out-of-order frame, the receiver immediately
|
||||
sends a pure ACK to help the sender converge.
|
||||
|
||||
### Why this is not the Two Generals Problem
|
||||
|
||||
We do not need mutual consensus. We need one-sided reliable delivery: the
|
||||
sender retransmits until it gets an ACK. If the ACK is lost, the sender
|
||||
retransmits the data; the receiver sees a duplicate, discards it, and re-ACKs.
|
||||
This converges in O(1) round trips. The only unsolvable case — the last frame
|
||||
before a permanent link death — is handled by `MAX_RETRIES` → `CLOSE`, which is
|
||||
correct: a dead link should kill the session.
|
||||
|
||||
## Payload field encodings
|
||||
|
||||
### MANIFEST payload
|
||||
@@ -129,8 +228,10 @@ the payload is exhausted. The number of entries is not carried explicitly.
|
||||
|
||||
### DATA payload
|
||||
|
||||
Raw application bytes. Up to 1480 bytes per frame. The `session_id` header
|
||||
field identifies which session the bytes belong to.
|
||||
`[ seq:4 ][ ack_seq:4 ][ raw bytes ]` — the `seq` and `ack_seq` fields are
|
||||
part of the DATA frame's extended header (between `payload_len` and the
|
||||
payload). The `payload_len` field counts only the raw bytes, not the seq/ack
|
||||
fields. Up to 1480 bytes of application data per frame.
|
||||
|
||||
### CLOSE payload
|
||||
|
||||
@@ -163,6 +264,7 @@ field identifies which session the bytes belong to.
|
||||
| 2 | connect_failed |
|
||||
| 3 | oversize |
|
||||
| 4 | unknown_session |
|
||||
| 5 | max_retries |
|
||||
|
||||
## Discovery flow
|
||||
|
||||
@@ -195,23 +297,27 @@ client server
|
||||
| OPEN_NAK { reason } |
|
||||
|<--------------------------------| (on failure)
|
||||
| |
|
||||
| DATA { session_id, bytes } |
|
||||
|<------------------------------->| DATA { session_id, bytes }
|
||||
| DATA { seq, ack_seq, bytes } |
|
||||
|<------------------------------->| DATA { seq, ack_seq, bytes }
|
||||
| |
|
||||
| CLOSE { session_id, reason? } |
|
||||
|<------------------------------->| (on EOF, RST, or error)
|
||||
|<------------------------------->| (on EOF, RST, or max_retries)
|
||||
| |
|
||||
```
|
||||
|
||||
- `session_id` is allocated by the server as a monotonically increasing u32
|
||||
(starting at 1) from an atomic counter. Collision by wraparound is ignored.
|
||||
- `seq` starts at 0 on both sides of each session and increments per DATA
|
||||
frame (including pure ACKs).
|
||||
- Either side may send `CLOSE`. The side receiving `CLOSE` tears down its half
|
||||
and stops emitting frames for that session.
|
||||
- The server's socket→tunnel pump reads `TcpStream` in 1480-byte chunks and
|
||||
emits one `DATA` frame per chunk. On `read` returning 0 (FIN) or an error,
|
||||
emits one DATA frame per chunk. On `read` returning 0 (FIN) or an error,
|
||||
it emits `CLOSE` and exits.
|
||||
- The server's tunnel→socket path writes `DATA` payloads to the `TcpStream`
|
||||
with `write_all`. On error it emits `CLOSE` and drops the session.
|
||||
- The server's tunnel→socket path delivers DATA payloads in seq order to the
|
||||
`TcpStream` with `write_all`. On error it emits `CLOSE` and drops the session.
|
||||
- A DATA frame is retransmitted if no ACK is received within 5 ms. After 10
|
||||
failed retransmits, the session is closed with `reason = max_retries`.
|
||||
|
||||
## Network test (PING/PONG)
|
||||
|
||||
@@ -235,3 +341,14 @@ client server
|
||||
compute RTT, average latency, jitter (mean absolute delta of consecutive
|
||||
RTTs), and drop rate (unanswered PINGs).
|
||||
- PING/PONG frames use `session_id = 0`; they are independent of TCP sessions.
|
||||
|
||||
## Version compatibility
|
||||
|
||||
- Version 1: no `seq`/`ack_seq` in DATA frames, no L2 reliability. Dropped
|
||||
DATA frames kill the TCP session.
|
||||
- Version 2: DATA frames carry `seq` + `ack_seq`, L2 retransmit. Dropped
|
||||
frames are recovered.
|
||||
|
||||
Version 2 is the current version. A receiver that sees `version != 2` rejects
|
||||
the frame. Both sides of a session must speak the same version; there is no
|
||||
negotiation.
|
||||
|
||||
Reference in New Issue
Block a user