hackmaster/Robovoice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.6: DHCP tunnel transport with NOP heartbeat protocol

Browse Source
This commit is contained in:
Mute
2026-08-12 00:29:54 +00:00
parent 4a58b94f35
commit e1739059d9
12 changed files with 743 additions and 528 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Server/NOTES.md
+242 -174
View File
@@ -1,202 +1,232 @@
# Server implementation notes
The STT server listens for TCP connections from Robovoice, captures audio
from a microphone when `on` is received, runs speech recognition (Moonshine),
and sends transcript messages back over the same connection.
The STT server listens on UDP port 67, receives NOP heartbeats and OFF
messages from Robovoice, captures audio from a microphone, runs speech
recognition (Moonshine), and sends transcript messages back to the client's
source address.
## Architecture
```
┌── on/off (TCP, newline-delimited JSON)
┌── HKMSTR <nonce> (broadcast, every 50ms)
Robovoice ──────────────►│
│ STT Server
Robovoice ◄──────────────┤
└── partial/final (TCP, newline-delimited JSON)
└── HKMSTR:P/F <text> (unicast)
```
The server:
1. Listens on a TCP port (e.g. 5210)
2. Accepts a connection from Robovoice
3. Reads lines: waits for `{"event":"on"}`
4. Records audio from the microphone
5. Waits for `{"event":"off"}` (or a timeout)
6. Runs STT on the captured audio
7. Sends `{"final":true,"text":"..."}`\n back over the connection
1. Listens on UDP :67
2. First `HKMSTR <nonce>` → start recording
3. `HKMSTR:OFF <nonce>` → stop recording, run STT
4. 150ms with no NOPs → stop recording, run STT (backstop)
5. Send `HKMSTR:F <text>` back to the client's source address:port
## Framing
Every message is a single JSON object on one line, terminated by `\n`. No
length prefix, no binary framing. Use `readline()` / `StreamReader.ReadLineAsync()`.
All messages are newline-terminated UTF-8 text. No JSON, no binary framing.
- `HKMSTR <nonce>` — NOP heartbeat (client → server)
- `HKMSTR:OFF <nonce>` — stop signal (client → server)
- `HKMSTR:P <text>` — partial transcript (server → client)
- `HKMSTR:F <text>` — final transcript (server → client)
The nonce is an incrementing integer for packet uniqueness only. Discard it.
## Python server with Moonshine
[Moonshine](https://github.com/usefulsensors/moonshine) is a lightweight ASR
model by Useful Sensors. Install with `pip install moonshine`.
```python
import socket
import json
import threading
import time
import numpy as np
import sounddevice as sd
import moonshine
LISTEN_PORT = 5210
LISTEN_PORT = 67
CLIENT_PORT = 68
SAMPLE_RATE = 16000
SILENCE_TIMEOUT = 0.150 # 150ms
model = moonshine.MoonshineModel(model="moonshine/base")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", LISTEN_PORT))
server.listen(1)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
sock.bind(("0.0.0.0", LISTEN_PORT))
print(f"STT server listening on :{LISTEN_PORT}")
recording = False
last_nop_time = 0
client_addr = None
audio_chunks = []
lock = threading.Lock()
def monitor_silence():
"""Backstop: stop recording if no NOPs for 150ms."""
global recording
while True:
time.sleep(0.01)
with lock:
if recording and (time.monotonic() - last_nop_time) > SILENCE_TIMEOUT:
recording = False
threading.Thread(target=process_audio, daemon=True).start()
threading.Thread(target=monitor_silence, daemon=True).start()
def process_audio():
global audio_chunks
with lock:
chunks = audio_chunks
audio_chunks = []
addr = client_addr
if not chunks:
return
audio = np.concatenate(chunks)
print(f"Captured {len(audio)/SAMPLE_RATE:.1f}s")
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Final: {text}")
reply = f"HKMSTR:F {text}\n".encode("utf-8")
sock.sendto(reply, addr)
else:
print("Empty transcript")
while True:
conn, addr = server.accept()
print(f"Client connected: {addr}")
data, addr = sock.recvfrom(4096)
text = data.decode("utf-8", errors="ignore").strip()
buf = ""
with conn:
while True:
data = conn.recv(4096).decode("utf-8")
if not data:
break
buf += data
if not text.startswith("HKMSTR"):
continue
while "\n" in buf:
line, buf = buf.split("\n", 1)
msg = json.loads(line)
if text.startswith("HKMSTR:OFF"):
with lock:
if recording:
recording = False
threading.Thread(target=process_audio, daemon=True).start()
continue
if msg.get("event") == "on":
print("PTT on — recording")
audio_chunks = []
if text.startswith("HKMSTR ") or text == "HKMSTR":
with lock:
client_addr = addr
last_nop_time = time.monotonic()
# Record until "off" or timeout
conn.settimeout(0.1)
while True:
try:
data2 = conn.recv(4096).decode("utf-8")
if not data2:
break
buf += data2
while "\n" in buf:
line2, buf = buf.split("\n", 1)
msg2 = json.loads(line2)
if msg2.get("event") == "off":
break
except socket.timeout:
pass
if not recording:
recording = True
audio_chunks = []
print(f"PTT on from {addr}")
chunk = sd.rec(int(SAMPLE_RATE * 0.1),
samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
conn.settimeout(None)
if not audio_chunks:
continue
audio = np.concatenate(audio_chunks)
print(f"Captured {len(audio)/SAMPLE_RATE:.1f}s")
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Transcript: {text}")
reply = json.dumps({"final": True, "text": text})
conn.sendall((reply + "\n").encode("utf-8"))
else:
print("Empty transcript")
# Capture 50ms of audio
chunk = sd.rec(int(SAMPLE_RATE * 0.05), samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
```
## Python server with streaming partials
For lower latency, send partial results while still recording:
For live feedback, send partials while recording:
```python
import socket
import json
import threading
import time
import numpy as np
import sounddevice as sd
import moonshine
LISTEN_PORT = 5210
LISTEN_PORT = 67
SAMPLE_RATE = 16000
CHUNK_DURATION = 0.5
SILENCE_TIMEOUT = 0.150
PARTIAL_INTERVAL = 0.5 # send partial every 500ms
model = moonshine.MoonshineModel(model="moonshine/base")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", LISTEN_PORT))
server.listen(1)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
sock.bind(("0.0.0.0", LISTEN_PORT))
print(f"STT server listening on :{LISTEN_PORT}")
recording = False
last_nop_time = 0
last_partial_time = 0
client_addr = None
audio_chunks = []
lock = threading.Lock()
def capture_and_maybe_partial():
global last_partial_time
with lock:
if not recording:
return
chunk = sd.rec(int(SAMPLE_RATE * 0.05), samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
now = time.monotonic()
if now - last_partial_time > PARTIAL_INTERVAL:
last_partial_time = now
partial_audio = np.concatenate(audio_chunks)
partial_text = moonshine.transcribe(model, partial_audio).strip()
if partial_text and client_addr:
reply = f"HKMSTR:P {partial_text}\n".encode("utf-8")
sock.sendto(reply, client_addr)
while True:
conn, addr = server.accept()
print(f"Client connected: {addr}")
buf = ""
data, addr = sock.recvfrom(4096)
text = data.decode("utf-8", errors="ignore").strip()
with conn:
while True:
data = conn.recv(4096).decode("utf-8")
if not data:
break
buf += data
if not text.startswith("HKMSTR"):
continue
while "\n" in buf:
line, buf = buf.split("\n", 1)
msg = json.loads(line)
if msg.get("event") != "on":
continue
print("PTT on — recording")
if text.startswith("HKMSTR:OFF"):
with lock:
if recording:
recording = False
chunks = audio_chunks
audio_chunks = []
while True:
try:
conn.settimeout(CHUNK_DURATION)
data2 = conn.recv(4096).decode("utf-8")
if not data2:
break
buf += data2
while "\n" in buf:
line2, buf = buf.split("\n", 1)
msg2 = json.loads(line2)
if msg2.get("event") == "off":
break
except socket.timeout:
pass
if chunks:
audio = np.concatenate(chunks)
final_text = moonshine.transcribe(model, audio).strip()
if final_text:
reply = f"HKMSTR:F {final_text}\n".encode("utf-8")
sock.sendto(reply, addr)
continue
chunk = sd.rec(int(SAMPLE_RATE * CHUNK_DURATION),
samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
if text.startswith("HKMSTR") and not text.startswith("HKMSTR:"):
with lock:
client_addr = addr
last_nop_time = time.monotonic()
# Send partial every few chunks
if len(audio_chunks) % 4 == 0:
partial_audio = np.concatenate(audio_chunks)
partial_text = moonshine.transcribe(model, partial_audio).strip()
if partial_text:
reply = json.dumps({"final": False, "text": partial_text})
conn.sendall((reply + "\n").encode("utf-8"))
if not recording:
recording = True
audio_chunks = []
last_partial_time = time.monotonic()
print(f"PTT on from {addr}")
conn.settimeout(None)
capture_and_maybe_partial()
if not audio_chunks:
continue
# Check silence timeout
with lock:
if recording and (time.monotonic() - last_nop_time) > SILENCE_TIMEOUT:
recording = False
chunks = audio_chunks
audio_chunks = []
audio = np.concatenate(audio_chunks)
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Final: {text}")
reply = json.dumps({"final": True, "text": text})
conn.sendall((reply + "\n").encode("utf-8"))
if 'chunks' in dir() and chunks:
audio = np.concatenate(chunks)
final_text = moonshine.transcribe(model, audio).strip()
if final_text:
reply = f"HKMSTR:F {final_text}\n".encode("utf-8")
sock.sendto(reply, addr)
```
## C# server skeleton
@@ -204,61 +234,99 @@ while True:
```csharp
using System.Net;
using System.Net.Sockets;
using System.Text.Json;
using System.Text;
var listener = new TcpListener(IPAddress.Any, 5210);
listener.Start();
var sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
sock.Bind(new IPEndPoint(IPAddress.Any, 67));
Console.WriteLine("STT server listening on :5210");
Console.WriteLine("STT server listening on :67");
byte[] buffer = new byte[4096];
EndPoint fromEp = new IPEndPoint(IPAddress.Any, 0);
bool recording = false;
DateTime lastNop = DateTime.MinValue;
List<float[]> audioChunks = new();
while (true)
{
var client = listener.AcceptTcpClient();
Console.WriteLine($"Client connected: {client.Client.RemoteEndPoint}");
using var stream = client.GetStream();
using var reader = new StreamReader(stream, Encoding.UTF8);
using var writer = new StreamWriter(stream, Encoding.UTF8) { AutoFlush = true };
string? line;
while ((line = reader.ReadLine()) is not null)
if (sock.Poll(50_000, SelectMode.SelectRead))
{
var msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
if (msg?["event"] != "on")
int received = sock.ReceiveFrom(buffer, ref fromEp);
string text = Encoding.UTF8.GetString(buffer, 0, received).TrimEnd('\n', '\r');
if (!text.StartsWith("HKMSTR"))
continue;
Console.WriteLine("PTT on — recording");
// Capture audio...
// Read until "off"
while ((line = reader.ReadLine()) is not null)
if (text.StartsWith("HKMSTR:OFF"))
{
msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
if (msg?["event"] == "off")
break;
if (recording)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
continue;
}
// Run STT...
string text = "recognized text here";
// NOP
lastNop = DateTime.UtcNow;
if (!recording)
{
recording = true;
audioChunks.Clear();
Console.WriteLine($"PTT on from {fromEp}");
}
var reply = JsonSerializer.Serialize(new { final = true, text });
writer.WriteLine(reply);
// Capture 50ms audio here...
// audioChunks.Add(capturedChunk);
// Check silence timeout
if (recording && (DateTime.UtcNow - lastNop).TotalMilliseconds > 150)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
}
else
{
// Timeout check even without incoming data
if (recording && (DateTime.UtcNow - lastNop).TotalMilliseconds > 150)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
}
}
void ProcessAndReply(List<float[]> chunks, EndPoint client)
{
if (chunks.Count == 0) return;
// Concatenate and run STT...
string text = "recognized text here";
if (!string.IsNullOrEmpty(text))
{
byte[] reply = Encoding.UTF8.GetBytes($"HKMSTR:F {text}\n");
sock.SendTo(reply, client);
}
}
```
## Tips
- **One connection per client:** Robovoice maintains a single persistent TCP
connection. The server should handle one client at a time (or track
multiple if needed).
- **Timeout:** implement a recording timeout in case the `off` message is
delayed or the client disconnects. 10–30 seconds is reasonable.
- **Partials:** optional but improve UX — Robovoice logs them so the user
sees live feedback. Only `final` triggers TTS.
- **Encoding:** always UTF-8. Every line is a UTF-8 JSON object terminated
by `\n`.
- **Reconnection:** Robovoice auto-reconnects every 3 seconds if the
connection drops. The server just needs to accept new connections.
- **Moonshine models:** `moonshine/base` (faster, less accurate) or
`moonshine/tiny` (fastest). Choose based on your hardware.
- **Reply address:** always reply to the source endpoint of the last NOP.
The client binds to a specific IP on :68.
- **Nonces:** discard them. They exist only to make each datagram unique.
Do not derive any meaning from nonce values.
- **Silence timeout:** 150ms = 3 missed NOPs at 50ms intervals. If you
change the NOP interval on the client, adjust this accordingly.
- **Partials:** optional. Send `HKMSTR:P <text>` while recording for live
feedback. Client logs them but only `HKMSTR:F` triggers TTS.
- **Broadcast only for C→S:** the WireGuard killswitch only allows
outbound broadcast to 255.255.255.255:67. Unicast from client won't pass.
- **Unicast OK for S→C:** the inbound WFP rule has no address restriction,
so unicast replies to :68 pass through.
- **Moonshine models:** `moonshine/base` or `moonshine/tiny`.
Server/PROTOCOL.md
+93 -52
View File
@@ -1,70 +1,111 @@
# Robovoice TCP STT Protocol
# Robovoice DHCP Tunnel Protocol
## Overview
Robovoice acts as a **client**: it connects to a remote STT server over TCP,
sends control messages when the user presses/releases the PTT key, and
receives transcript messages back. The STT server captures audio from a
microphone, runs speech recognition (Moonshine), and sends transcripts back
over the same connection.
Robovoice communicates with a remote STT server by tunneling through the
DHCP UDP ports (68→67). This exploits a common killswitch exception: VPN
software (e.g. WireGuard) blocks all traffic except DHCP, which is allowed
for network connectivity maintenance.
```
[Robovoice client] --TCP--> [STT server :5210]
│ │
├── {"event":"on"}\n ──────►│
│ ├── capture audio
├── {"event":"off"}\n ──────►│
│ ├── run STT
│◄── {"final":true,...}\n ──┤
[Robovoice client] --broadcast UDP :68→:67--> [STT server]
[Robovoice client] <--unicast UDP :67→:68-- [STT server]
```
The client broadcasts NOP heartbeats while PTT is held. The server starts
recording on the first NOP and stops when it receives OFF or when 150ms
pass with no NOPs.
## Transport
- **Protocol:** TCP (reliable, ordered, connection-oriented)
- **Server endpoint:** configurable in Robovoice UI (default `127.0.0.1:5210`)
- **Framing:** newline-delimited JSON (NDJSON) — each message is a single
UTF-8 JSON object terminated by `\n`
- **Auto-reconnect:** if the connection drops, Robovoice retries every 3
seconds until the server is available
- **Protocol:** UDP (connectionless, unreliable)
- **Client → Server:** broadcast, source port 68, dest port 67
- **Server → Client:** unicast, source port 67, dest port 68
- **Client binds:** to a specific LAN interface IP on port 68 (with
`SO_REUSEADDR` to coexist with the Windows DHCP service)
- **No connection state** — purely fire-and-forget datagrams
## Control messages (client → server)
## Wire format
Sent by Robovoice when the user presses/releases the PTT key.
All messages are plain text, newline-terminated (`\n`). Every message starts
with the 6-byte magic `HKMSTR` to distinguish our traffic from real DHCP.
```json
{"event": "on"}
### Client → Server
**NOP (heartbeat while PTT held):**
```
HKMSTR <nonce>\n
```
Sent every 50ms while PTT is held. The nonce is an incrementing unsigned
integer that makes each datagram unique. The server discards it — it's
purely for packet uniqueness, not for any protocol logic.
**OFF (PTT released):**
```
HKMSTR:OFF <nonce>\n
```
Sent once when PTT is released. This is the fast-stop signal. If lost, the
150ms timeout acts as a backstop.
### Server → Client
**Partial transcript:**
```
HKMSTR:P <text>\n
```
Intermediate recognition result. Fire-and-forget. Client logs it but does
not act on it.
**Final transcript:**
```
HKMSTR:F <text>\n
```
Complete utterance. Client feeds this to the TTS engine.
## Server state machine
```
┌──────────────────────────────────────────┐
│ │
▼ │
┌──────────┐ first NOP ┌──────────────┐ │
│ IDLE │ ──────────► │ RECORDING │ │
└──────────┘ └──────────────┘ │
│ │ │
OFF │ │ 150ms │
recv'd │ │ silence │
▼ ▼ │
┌─────────────┐ │
│ PROCESSING │ │
└─────────────┘ │
│ │
STT │ │
done │ │
▼ │
send HKMSTR:F ─────────┘
```
```json
{"event": "off"}
```
- **IDLE → RECORDING:** first NOP received, start mic capture
- **RECORDING → PROCESSING:** OFF received, OR 150ms since last NOP
- **PROCESSING → IDLE:** STT done, send `HKMSTR:F <text>`
| Field | Type | Description |
|---------|--------|------------------------------------|
| `event` | string | `"on"` (PTT pressed) or `"off"` (PTT released) |
## Timing
## Transcript messages (server → client)
| Parameter | Value | Purpose |
|-----------|-------|---------|
| NOP interval | 50ms | Heartbeat frequency while PTT held |
| Silence timeout | 150ms | Stop recording if no NOPs (3 missed = lost OFF) |
| NOP bandwidth | ~20 msg/s × ~20 bytes | ~400 bytes/s — negligible |
Sent by the server back to Robovoice over the same TCP connection.
## Why this works
```json
{"final": false, "text": "hello world"}
```
```json
{"final": true, "text": "hello world how are you"}
```
| Field | Type | Required | Description |
|---------|---------|----------|--------------------------------------------------|
| `final` | bool | yes | `true` = final result, `false` = partial |
| `text` | string | yes | The transcript text (may be empty for partials) |
### Semantics
- **`final: false`** — intermediate recognition result (partial). Robovoice
logs these but does not act on them (only `final` triggers TTS).
- **`final: true`** — complete utterance. Robovoice feeds this to the TTS
engine and speaks it.
Malformed JSON or unknown field values are silently dropped by the client.
1. **Outbound broadcast `:68→:67` to `255.255.255.255`** passes the
WireGuard WFP killswitch (DHCP exception matches this exact pattern)
2. **Inbound `:67→:68`** has no address restriction in the WFP rule, so
unicast replies pass through
3. **Binding to a specific interface IP** (not `0.0.0.0`) wins unicast
delivery over the Windows DHCP client service
4. **NOP spam** ensures the ON message gets through even at 5% packet loss
(3 consecutive NOPs = ~0.01% drop probability)
5. **150ms timeout** is the backstop for lost OFF — at 50ms intervals, 3
consecutive NOPs must all be lost to false-stop