107 lines
4.1 KiB
Markdown
107 lines
4.1 KiB
Markdown
# Robovoice STT Protocol
|
|
|
|
## Overview
|
|
|
|
Robovoice connects to the STT server over TCP (typically through a gatuna
|
|
L2 tunnel). The server captures audio, runs Moonshine STT, and sends
|
|
transcript segments back. The client pre-synthesizes TTS on segments and
|
|
plays audio on final.
|
|
|
|
```
|
|
[Robovoice client] --TCP--> [STT server 127.0.0.1:6996]
|
|
│ │
|
|
├── ON <session>\n ────────►│ (abort old, start new session)
|
|
├── OFF <session>\n ────────►│ (stop, final STT pass)
|
|
│◄── P <session> <text>\n ──┤ (completed segment)
|
|
│◄── F <session> <text>\n ──┤ (all done; text may be empty)
|
|
```
|
|
|
|
## Transport
|
|
|
|
- **Protocol:** TCP (reliable, ordered, connection-oriented)
|
|
- **Server:** `127.0.0.1:6996` (hardcoded loopback)
|
|
- **Framing:** newline-delimited text (`\n`), UTF-8
|
|
- **Auto-reconnect:** client retries every 3s if connection drops
|
|
|
|
## Wire format
|
|
|
|
### Client → Server
|
|
|
|
**ON (PTT pressed):**
|
|
```
|
|
ON <session>\n
|
|
```
|
|
Starts a new STT session. The server aborts any active session and starts
|
|
recording. `<session>` is an incrementing unsigned integer chosen by the
|
|
client. Replies from the server echo this session ID.
|
|
|
|
**OFF (PTT released):**
|
|
```
|
|
OFF <session>\n
|
|
```
|
|
Stops the session. The server does a final STT pass on remaining audio and
|
|
sends any new segments followed by `F`.
|
|
|
|
### Server → Client
|
|
|
|
**Segment (completed VAD segment):**
|
|
```
|
|
P <session> <text>\n
|
|
```
|
|
A completed, VAD-separated utterance segment. The client starts TTS
|
|
synthesis immediately and buffers the audio (does not play yet).
|
|
|
|
**Final (all done):**
|
|
```
|
|
F <session> <text>\n
|
|
```
|
|
Signals all segments have been sent. `<text>` may be empty (`F <session>\n`).
|
|
Triggers playback of all buffered audio on the client. If text is non-empty,
|
|
the client synthesizes it before playing.
|
|
|
|
## Session IDs
|
|
|
|
- Client increments session ID on each PTT press
|
|
- Server echoes the session ID in all replies for that session
|
|
- Client drops any reply with a stale session ID (handles the race where
|
|
stale segments from an aborted session are still in the TCP buffer)
|
|
- Server aborts old session on receiving `ON` with a new session ID
|
|
|
|
## Client playback model
|
|
|
|
1. `P` arrives → start TTS synthesis immediately, buffer audio (don't play)
|
|
2. More `P` arrive → keep synthesizing and buffering
|
|
3. `F` arrives → play all buffered audio immediately
|
|
4. PTT pressed → flush: stop playback, cancel synthesis, clear buffers
|
|
|
|
The purpose of pre-synthesis is to minimize latency between PTT release
|
|
and audio playback. By the time `F` arrives, audio is already buffered.
|
|
|
|
## Server state machine
|
|
|
|
```
|
|
┌──────────┐ ON <session> ┌──────────────┐
|
|
│ IDLE │ ──────────────► │ RECORDING │
|
|
└──────────┘ └──────────────┘
|
|
│ │
|
|
OFF │ │
|
|
recv'd │ │
|
|
▼ │
|
|
┌─────────────┐
|
|
│ PROCESSING │
|
|
└─────────────┘
|
|
│
|
|
send │
|
|
P/F │
|
|
▼
|
|
back to IDLE
|
|
```
|
|
|
|
- **IDLE → RECORDING:** `ON <session>` received, start mic capture
|
|
- **RECORDING:** Moonshine streaming produces completed segments → send `P`
|
|
- **RECORDING → PROCESSING:** `OFF <session>` received
|
|
- **PROCESSING → IDLE:** final STT pass, send remaining `P` + `F`
|
|
|
|
If `ON` arrives while recording, the current session is aborted (no final
|
|
flush) and a new session starts immediately.
|