hackmaster/Robovoice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.6: DHCP tunnel transport with NOP heartbeat protocol

Browse Source
This commit is contained in:
Mute
2026-08-12 00:29:54 +00:00
parent 4a58b94f35
commit e1739059d9
12 changed files with 743 additions and 528 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Server/NOTES.md
+242 -174
View File
@@ -1,202 +1,232 @@
# Server implementation notes
The STT server listens for TCP connections from Robovoice, captures audio
from a microphone when `on` is received, runs speech recognition (Moonshine),
and sends transcript messages back over the same connection.
The STT server listens on UDP port 67, receives NOP heartbeats and OFF
messages from Robovoice, captures audio from a microphone, runs speech
recognition (Moonshine), and sends transcript messages back to the client's
source address.
## Architecture
```
┌── on/off (TCP, newline-delimited JSON)
┌── HKMSTR <nonce> (broadcast, every 50ms)
Robovoice ──────────────►│
│ STT Server
Robovoice ◄──────────────┤
└── partial/final (TCP, newline-delimited JSON)
└── HKMSTR:P/F <text> (unicast)
```
The server:
1. Listens on a TCP port (e.g. 5210)
2. Accepts a connection from Robovoice
3. Reads lines: waits for `{"event":"on"}`
4. Records audio from the microphone
5. Waits for `{"event":"off"}` (or a timeout)
6. Runs STT on the captured audio
7. Sends `{"final":true,"text":"..."}`\n back over the connection
1. Listens on UDP :67
2. First `HKMSTR <nonce>` → start recording
3. `HKMSTR:OFF <nonce>` → stop recording, run STT
4. 150ms with no NOPs → stop recording, run STT (backstop)
5. Send `HKMSTR:F <text>` back to the client's source address:port
## Framing
Every message is a single JSON object on one line, terminated by `\n`. No
length prefix, no binary framing. Use `readline()` / `StreamReader.ReadLineAsync()`.
All messages are newline-terminated UTF-8 text. No JSON, no binary framing.
- `HKMSTR <nonce>` — NOP heartbeat (client → server)
- `HKMSTR:OFF <nonce>` — stop signal (client → server)
- `HKMSTR:P <text>` — partial transcript (server → client)
- `HKMSTR:F <text>` — final transcript (server → client)
The nonce is an incrementing integer for packet uniqueness only. Discard it.
## Python server with Moonshine
[Moonshine](https://github.com/usefulsensors/moonshine) is a lightweight ASR
model by Useful Sensors. Install with `pip install moonshine`.
```python
import socket
import json
import threading
import time
import numpy as np
import sounddevice as sd
import moonshine
LISTEN_PORT = 5210
LISTEN_PORT = 67
CLIENT_PORT = 68
SAMPLE_RATE = 16000
SILENCE_TIMEOUT = 0.150 # 150ms
model = moonshine.MoonshineModel(model="moonshine/base")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", LISTEN_PORT))
server.listen(1)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
sock.bind(("0.0.0.0", LISTEN_PORT))
print(f"STT server listening on :{LISTEN_PORT}")
recording = False
last_nop_time = 0
client_addr = None
audio_chunks = []
lock = threading.Lock()
def monitor_silence():
"""Backstop: stop recording if no NOPs for 150ms."""
global recording
while True:
time.sleep(0.01)
with lock:
if recording and (time.monotonic() - last_nop_time) > SILENCE_TIMEOUT:
recording = False
threading.Thread(target=process_audio, daemon=True).start()
threading.Thread(target=monitor_silence, daemon=True).start()
def process_audio():
global audio_chunks
with lock:
chunks = audio_chunks
audio_chunks = []
addr = client_addr
if not chunks:
return
audio = np.concatenate(chunks)
print(f"Captured {len(audio)/SAMPLE_RATE:.1f}s")
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Final: {text}")
reply = f"HKMSTR:F {text}\n".encode("utf-8")
sock.sendto(reply, addr)
else:
print("Empty transcript")
while True:
conn, addr = server.accept()
print(f"Client connected: {addr}")
data, addr = sock.recvfrom(4096)
text = data.decode("utf-8", errors="ignore").strip()
buf = ""
with conn:
while True:
data = conn.recv(4096).decode("utf-8")
if not data:
break
buf += data
if not text.startswith("HKMSTR"):
continue
while "\n" in buf:
line, buf = buf.split("\n", 1)
msg = json.loads(line)
if text.startswith("HKMSTR:OFF"):
with lock:
if recording:
recording = False
threading.Thread(target=process_audio, daemon=True).start()
continue
if msg.get("event") == "on":
print("PTT on — recording")
audio_chunks = []
if text.startswith("HKMSTR ") or text == "HKMSTR":
with lock:
client_addr = addr
last_nop_time = time.monotonic()
# Record until "off" or timeout
conn.settimeout(0.1)
while True:
try:
data2 = conn.recv(4096).decode("utf-8")
if not data2:
break
buf += data2
while "\n" in buf:
line2, buf = buf.split("\n", 1)
msg2 = json.loads(line2)
if msg2.get("event") == "off":
break
except socket.timeout:
pass
if not recording:
recording = True
audio_chunks = []
print(f"PTT on from {addr}")
chunk = sd.rec(int(SAMPLE_RATE * 0.1),
samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
conn.settimeout(None)
if not audio_chunks:
continue
audio = np.concatenate(audio_chunks)
print(f"Captured {len(audio)/SAMPLE_RATE:.1f}s")
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Transcript: {text}")
reply = json.dumps({"final": True, "text": text})
conn.sendall((reply + "\n").encode("utf-8"))
else:
print("Empty transcript")
# Capture 50ms of audio
chunk = sd.rec(int(SAMPLE_RATE * 0.05), samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
```
## Python server with streaming partials
For lower latency, send partial results while still recording:
For live feedback, send partials while recording:
```python
import socket
import json
import threading
import time
import numpy as np
import sounddevice as sd
import moonshine
LISTEN_PORT = 5210
LISTEN_PORT = 67
SAMPLE_RATE = 16000
CHUNK_DURATION = 0.5
SILENCE_TIMEOUT = 0.150
PARTIAL_INTERVAL = 0.5 # send partial every 500ms
model = moonshine.MoonshineModel(model="moonshine/base")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", LISTEN_PORT))
server.listen(1)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
sock.bind(("0.0.0.0", LISTEN_PORT))
print(f"STT server listening on :{LISTEN_PORT}")
recording = False
last_nop_time = 0
last_partial_time = 0
client_addr = None
audio_chunks = []
lock = threading.Lock()
def capture_and_maybe_partial():
global last_partial_time
with lock:
if not recording:
return
chunk = sd.rec(int(SAMPLE_RATE * 0.05), samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
now = time.monotonic()
if now - last_partial_time > PARTIAL_INTERVAL:
last_partial_time = now
partial_audio = np.concatenate(audio_chunks)
partial_text = moonshine.transcribe(model, partial_audio).strip()
if partial_text and client_addr:
reply = f"HKMSTR:P {partial_text}\n".encode("utf-8")
sock.sendto(reply, client_addr)
while True:
conn, addr = server.accept()
print(f"Client connected: {addr}")
buf = ""
data, addr = sock.recvfrom(4096)
text = data.decode("utf-8", errors="ignore").strip()
with conn:
while True:
data = conn.recv(4096).decode("utf-8")
if not data:
break
buf += data
if not text.startswith("HKMSTR"):
continue
while "\n" in buf:
line, buf = buf.split("\n", 1)
msg = json.loads(line)
if msg.get("event") != "on":
continue
print("PTT on — recording")
if text.startswith("HKMSTR:OFF"):
with lock:
if recording:
recording = False
chunks = audio_chunks
audio_chunks = []
while True:
try:
conn.settimeout(CHUNK_DURATION)
data2 = conn.recv(4096).decode("utf-8")
if not data2:
break
buf += data2
while "\n" in buf:
line2, buf = buf.split("\n", 1)
msg2 = json.loads(line2)
if msg2.get("event") == "off":
break
except socket.timeout:
pass
if chunks:
audio = np.concatenate(chunks)
final_text = moonshine.transcribe(model, audio).strip()
if final_text:
reply = f"HKMSTR:F {final_text}\n".encode("utf-8")
sock.sendto(reply, addr)
continue
chunk = sd.rec(int(SAMPLE_RATE * CHUNK_DURATION),
samplerate=SAMPLE_RATE,
channels=1, dtype="float32")
sd.wait()
audio_chunks.append(chunk.flatten())
if text.startswith("HKMSTR") and not text.startswith("HKMSTR:"):
with lock:
client_addr = addr
last_nop_time = time.monotonic()
# Send partial every few chunks
if len(audio_chunks) % 4 == 0:
partial_audio = np.concatenate(audio_chunks)
partial_text = moonshine.transcribe(model, partial_audio).strip()
if partial_text:
reply = json.dumps({"final": False, "text": partial_text})
conn.sendall((reply + "\n").encode("utf-8"))
if not recording:
recording = True
audio_chunks = []
last_partial_time = time.monotonic()
print(f"PTT on from {addr}")
conn.settimeout(None)
capture_and_maybe_partial()
if not audio_chunks:
continue
# Check silence timeout
with lock:
if recording and (time.monotonic() - last_nop_time) > SILENCE_TIMEOUT:
recording = False
chunks = audio_chunks
audio_chunks = []
audio = np.concatenate(audio_chunks)
text = moonshine.transcribe(model, audio).strip()
if text:
print(f"Final: {text}")
reply = json.dumps({"final": True, "text": text})
conn.sendall((reply + "\n").encode("utf-8"))
if 'chunks' in dir() and chunks:
audio = np.concatenate(chunks)
final_text = moonshine.transcribe(model, audio).strip()
if final_text:
reply = f"HKMSTR:F {final_text}\n".encode("utf-8")
sock.sendto(reply, addr)
```
## C# server skeleton
@@ -204,61 +234,99 @@ while True:
```csharp
using System.Net;
using System.Net.Sockets;
using System.Text.Json;
using System.Text;
var listener = new TcpListener(IPAddress.Any, 5210);
listener.Start();
var sock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
sock.Bind(new IPEndPoint(IPAddress.Any, 67));
Console.WriteLine("STT server listening on :5210");
Console.WriteLine("STT server listening on :67");
byte[] buffer = new byte[4096];
EndPoint fromEp = new IPEndPoint(IPAddress.Any, 0);
bool recording = false;
DateTime lastNop = DateTime.MinValue;
List<float[]> audioChunks = new();
while (true)
{
var client = listener.AcceptTcpClient();
Console.WriteLine($"Client connected: {client.Client.RemoteEndPoint}");
using var stream = client.GetStream();
using var reader = new StreamReader(stream, Encoding.UTF8);
using var writer = new StreamWriter(stream, Encoding.UTF8) { AutoFlush = true };
string? line;
while ((line = reader.ReadLine()) is not null)
if (sock.Poll(50_000, SelectMode.SelectRead))
{
var msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
if (msg?["event"] != "on")
int received = sock.ReceiveFrom(buffer, ref fromEp);
string text = Encoding.UTF8.GetString(buffer, 0, received).TrimEnd('\n', '\r');
if (!text.StartsWith("HKMSTR"))
continue;
Console.WriteLine("PTT on — recording");
// Capture audio...
// Read until "off"
while ((line = reader.ReadLine()) is not null)
if (text.StartsWith("HKMSTR:OFF"))
{
msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
if (msg?["event"] == "off")
break;
if (recording)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
continue;
}
// Run STT...
string text = "recognized text here";
// NOP
lastNop = DateTime.UtcNow;
if (!recording)
{
recording = true;
audioChunks.Clear();
Console.WriteLine($"PTT on from {fromEp}");
}
var reply = JsonSerializer.Serialize(new { final = true, text });
writer.WriteLine(reply);
// Capture 50ms audio here...
// audioChunks.Add(capturedChunk);
// Check silence timeout
if (recording && (DateTime.UtcNow - lastNop).TotalMilliseconds > 150)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
}
else
{
// Timeout check even without incoming data
if (recording && (DateTime.UtcNow - lastNop).TotalMilliseconds > 150)
{
recording = false;
ProcessAndReply(audioChunks, fromEp);
audioChunks.Clear();
}
}
}
void ProcessAndReply(List<float[]> chunks, EndPoint client)
{
if (chunks.Count == 0) return;
// Concatenate and run STT...
string text = "recognized text here";
if (!string.IsNullOrEmpty(text))
{
byte[] reply = Encoding.UTF8.GetBytes($"HKMSTR:F {text}\n");
sock.SendTo(reply, client);
}
}
```
## Tips
- **One connection per client:** Robovoice maintains a single persistent TCP
connection. The server should handle one client at a time (or track
multiple if needed).
- **Timeout:** implement a recording timeout in case the `off` message is
delayed or the client disconnects. 10–30 seconds is reasonable.
- **Partials:** optional but improve UX — Robovoice logs them so the user
sees live feedback. Only `final` triggers TTS.
- **Encoding:** always UTF-8. Every line is a UTF-8 JSON object terminated
by `\n`.
- **Reconnection:** Robovoice auto-reconnects every 3 seconds if the
connection drops. The server just needs to accept new connections.
- **Moonshine models:** `moonshine/base` (faster, less accurate) or
`moonshine/tiny` (fastest). Choose based on your hardware.
- **Reply address:** always reply to the source endpoint of the last NOP.
The client binds to a specific IP on :68.
- **Nonces:** discard them. They exist only to make each datagram unique.
Do not derive any meaning from nonce values.
- **Silence timeout:** 150ms = 3 missed NOPs at 50ms intervals. If you
change the NOP interval on the client, adjust this accordingly.
- **Partials:** optional. Send `HKMSTR:P <text>` while recording for live
feedback. Client logs them but only `HKMSTR:F` triggers TTS.
- **Broadcast only for C→S:** the WireGuard killswitch only allows
outbound broadcast to 255.255.255.255:67. Unicast from client won't pass.
- **Unicast OK for S→C:** the inbound WFP rule has no address restriction,
so unicast replies to :68 pass through.
- **Moonshine models:** `moonshine/base` or `moonshine/tiny`.