v0.5: TCP STT client with PTT on/off, server endpoint in config + UI
This commit is contained in:
+264
@@ -0,0 +1,264 @@
|
||||
# Server implementation notes
|
||||
|
||||
The STT server listens for TCP connections from Robovoice, captures audio
|
||||
from a microphone when `on` is received, runs speech recognition (Moonshine),
|
||||
and sends transcript messages back over the same connection.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌── on/off (TCP, newline-delimited JSON)
|
||||
Robovoice ──────────────►│
|
||||
│ STT Server
|
||||
Robovoice ◄──────────────┤
|
||||
└── partial/final (TCP, newline-delimited JSON)
|
||||
```
|
||||
|
||||
The server:
|
||||
1. Listens on a TCP port (e.g. 5210)
|
||||
2. Accepts a connection from Robovoice
|
||||
3. Reads lines: waits for `{"event":"on"}`
|
||||
4. Records audio from the microphone
|
||||
5. Waits for `{"event":"off"}` (or a timeout)
|
||||
6. Runs STT on the captured audio
|
||||
7. Sends `{"final":true,"text":"..."}`\n back over the connection
|
||||
|
||||
## Framing
|
||||
|
||||
Every message is a single JSON object on one line, terminated by `\n`. No
|
||||
length prefix, no binary framing. Use `readline()` / `StreamReader.ReadLineAsync()`.
|
||||
|
||||
## Python server with Moonshine
|
||||
|
||||
[Moonshine](https://github.com/usefulsensors/moonshine) is a lightweight ASR
|
||||
model by Useful Sensors. Install with `pip install moonshine`.
|
||||
|
||||
```python
|
||||
import socket
|
||||
import json
|
||||
import numpy as np
|
||||
import sounddevice as sd
|
||||
import moonshine
|
||||
|
||||
LISTEN_PORT = 5210
|
||||
SAMPLE_RATE = 16000
|
||||
|
||||
model = moonshine.MoonshineModel(model="moonshine/base")
|
||||
|
||||
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
server.bind(("0.0.0.0", LISTEN_PORT))
|
||||
server.listen(1)
|
||||
|
||||
print(f"STT server listening on :{LISTEN_PORT}")
|
||||
|
||||
while True:
|
||||
conn, addr = server.accept()
|
||||
print(f"Client connected: {addr}")
|
||||
|
||||
buf = ""
|
||||
with conn:
|
||||
while True:
|
||||
data = conn.recv(4096).decode("utf-8")
|
||||
if not data:
|
||||
break
|
||||
buf += data
|
||||
|
||||
while "\n" in buf:
|
||||
line, buf = buf.split("\n", 1)
|
||||
msg = json.loads(line)
|
||||
|
||||
if msg.get("event") == "on":
|
||||
print("PTT on — recording")
|
||||
audio_chunks = []
|
||||
|
||||
# Record until "off" or timeout
|
||||
conn.settimeout(0.1)
|
||||
while True:
|
||||
try:
|
||||
data2 = conn.recv(4096).decode("utf-8")
|
||||
if not data2:
|
||||
break
|
||||
buf += data2
|
||||
while "\n" in buf:
|
||||
line2, buf = buf.split("\n", 1)
|
||||
msg2 = json.loads(line2)
|
||||
if msg2.get("event") == "off":
|
||||
break
|
||||
except socket.timeout:
|
||||
pass
|
||||
|
||||
chunk = sd.rec(int(SAMPLE_RATE * 0.1),
|
||||
samplerate=SAMPLE_RATE,
|
||||
channels=1, dtype="float32")
|
||||
sd.wait()
|
||||
audio_chunks.append(chunk.flatten())
|
||||
|
||||
conn.settimeout(None)
|
||||
|
||||
if not audio_chunks:
|
||||
continue
|
||||
|
||||
audio = np.concatenate(audio_chunks)
|
||||
print(f"Captured {len(audio)/SAMPLE_RATE:.1f}s")
|
||||
|
||||
text = moonshine.transcribe(model, audio).strip()
|
||||
|
||||
if text:
|
||||
print(f"Transcript: {text}")
|
||||
reply = json.dumps({"final": True, "text": text})
|
||||
conn.sendall((reply + "\n").encode("utf-8"))
|
||||
else:
|
||||
print("Empty transcript")
|
||||
```
|
||||
|
||||
## Python server with streaming partials
|
||||
|
||||
For lower latency, send partial results while still recording:
|
||||
|
||||
```python
|
||||
import socket
|
||||
import json
|
||||
import numpy as np
|
||||
import sounddevice as sd
|
||||
import moonshine
|
||||
|
||||
LISTEN_PORT = 5210
|
||||
SAMPLE_RATE = 16000
|
||||
CHUNK_DURATION = 0.5
|
||||
|
||||
model = moonshine.MoonshineModel(model="moonshine/base")
|
||||
|
||||
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
server.bind(("0.0.0.0", LISTEN_PORT))
|
||||
server.listen(1)
|
||||
|
||||
print(f"STT server listening on :{LISTEN_PORT}")
|
||||
|
||||
while True:
|
||||
conn, addr = server.accept()
|
||||
print(f"Client connected: {addr}")
|
||||
buf = ""
|
||||
|
||||
with conn:
|
||||
while True:
|
||||
data = conn.recv(4096).decode("utf-8")
|
||||
if not data:
|
||||
break
|
||||
buf += data
|
||||
|
||||
while "\n" in buf:
|
||||
line, buf = buf.split("\n", 1)
|
||||
msg = json.loads(line)
|
||||
|
||||
if msg.get("event") != "on":
|
||||
continue
|
||||
|
||||
print("PTT on — recording")
|
||||
audio_chunks = []
|
||||
|
||||
while True:
|
||||
try:
|
||||
conn.settimeout(CHUNK_DURATION)
|
||||
data2 = conn.recv(4096).decode("utf-8")
|
||||
if not data2:
|
||||
break
|
||||
buf += data2
|
||||
while "\n" in buf:
|
||||
line2, buf = buf.split("\n", 1)
|
||||
msg2 = json.loads(line2)
|
||||
if msg2.get("event") == "off":
|
||||
break
|
||||
except socket.timeout:
|
||||
pass
|
||||
|
||||
chunk = sd.rec(int(SAMPLE_RATE * CHUNK_DURATION),
|
||||
samplerate=SAMPLE_RATE,
|
||||
channels=1, dtype="float32")
|
||||
sd.wait()
|
||||
audio_chunks.append(chunk.flatten())
|
||||
|
||||
# Send partial every few chunks
|
||||
if len(audio_chunks) % 4 == 0:
|
||||
partial_audio = np.concatenate(audio_chunks)
|
||||
partial_text = moonshine.transcribe(model, partial_audio).strip()
|
||||
if partial_text:
|
||||
reply = json.dumps({"final": False, "text": partial_text})
|
||||
conn.sendall((reply + "\n").encode("utf-8"))
|
||||
|
||||
conn.settimeout(None)
|
||||
|
||||
if not audio_chunks:
|
||||
continue
|
||||
|
||||
audio = np.concatenate(audio_chunks)
|
||||
text = moonshine.transcribe(model, audio).strip()
|
||||
|
||||
if text:
|
||||
print(f"Final: {text}")
|
||||
reply = json.dumps({"final": True, "text": text})
|
||||
conn.sendall((reply + "\n").encode("utf-8"))
|
||||
```
|
||||
|
||||
## C# server skeleton
|
||||
|
||||
```csharp
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using System.Text.Json;
|
||||
|
||||
var listener = new TcpListener(IPAddress.Any, 5210);
|
||||
listener.Start();
|
||||
|
||||
Console.WriteLine("STT server listening on :5210");
|
||||
|
||||
while (true)
|
||||
{
|
||||
var client = listener.AcceptTcpClient();
|
||||
Console.WriteLine($"Client connected: {client.Client.RemoteEndPoint}");
|
||||
|
||||
using var stream = client.GetStream();
|
||||
using var reader = new StreamReader(stream, Encoding.UTF8);
|
||||
using var writer = new StreamWriter(stream, Encoding.UTF8) { AutoFlush = true };
|
||||
|
||||
string? line;
|
||||
while ((line = reader.ReadLine()) is not null)
|
||||
{
|
||||
var msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
|
||||
if (msg?["event"] != "on")
|
||||
continue;
|
||||
|
||||
Console.WriteLine("PTT on — recording");
|
||||
// Capture audio...
|
||||
|
||||
// Read until "off"
|
||||
while ((line = reader.ReadLine()) is not null)
|
||||
{
|
||||
msg = JsonSerializer.Deserialize<Dictionary<string, string>>(line);
|
||||
if (msg?["event"] == "off")
|
||||
break;
|
||||
}
|
||||
|
||||
// Run STT...
|
||||
string text = "recognized text here";
|
||||
|
||||
var reply = JsonSerializer.Serialize(new { final = true, text });
|
||||
writer.WriteLine(reply);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- **One connection per client:** Robovoice maintains a single persistent TCP
|
||||
connection. The server should handle one client at a time (or track
|
||||
multiple if needed).
|
||||
- **Timeout:** implement a recording timeout in case the `off` message is
|
||||
delayed or the client disconnects. 10–30 seconds is reasonable.
|
||||
- **Partials:** optional but improve UX — Robovoice logs them so the user
|
||||
sees live feedback. Only `final` triggers TTS.
|
||||
- **Encoding:** always UTF-8. Every line is a UTF-8 JSON object terminated
|
||||
by `\n`.
|
||||
- **Reconnection:** Robovoice auto-reconnects every 3 seconds if the
|
||||
connection drops. The server just needs to accept new connections.
|
||||
- **Moonshine models:** `moonshine/base` (faster, less accurate) or
|
||||
`moonshine/tiny` (fastest). Choose based on your hardware.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Robovoice TCP STT Protocol
|
||||
|
||||
## Overview
|
||||
|
||||
Robovoice acts as a **client**: it connects to a remote STT server over TCP,
|
||||
sends control messages when the user presses/releases the PTT key, and
|
||||
receives transcript messages back. The STT server captures audio from a
|
||||
microphone, runs speech recognition (Moonshine), and sends transcripts back
|
||||
over the same connection.
|
||||
|
||||
```
|
||||
[Robovoice client] --TCP--> [STT server :5210]
|
||||
│ │
|
||||
├── {"event":"on"}\n ──────►│
|
||||
│ ├── capture audio
|
||||
├── {"event":"off"}\n ──────►│
|
||||
│ ├── run STT
|
||||
│◄── {"final":true,...}\n ──┤
|
||||
```
|
||||
|
||||
## Transport
|
||||
|
||||
- **Protocol:** TCP (reliable, ordered, connection-oriented)
|
||||
- **Server endpoint:** configurable in Robovoice UI (default `127.0.0.1:5210`)
|
||||
- **Framing:** newline-delimited JSON (NDJSON) — each message is a single
|
||||
UTF-8 JSON object terminated by `\n`
|
||||
- **Auto-reconnect:** if the connection drops, Robovoice retries every 3
|
||||
seconds until the server is available
|
||||
|
||||
## Control messages (client → server)
|
||||
|
||||
Sent by Robovoice when the user presses/releases the PTT key.
|
||||
|
||||
```json
|
||||
{"event": "on"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"event": "off"}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|---------|--------|------------------------------------|
|
||||
| `event` | string | `"on"` (PTT pressed) or `"off"` (PTT released) |
|
||||
|
||||
## Transcript messages (server → client)
|
||||
|
||||
Sent by the server back to Robovoice over the same TCP connection.
|
||||
|
||||
```json
|
||||
{"final": false, "text": "hello world"}
|
||||
```
|
||||
|
||||
```json
|
||||
{"final": true, "text": "hello world how are you"}
|
||||
```
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|---------|---------|----------|--------------------------------------------------|
|
||||
| `final` | bool | yes | `true` = final result, `false` = partial |
|
||||
| `text` | string | yes | The transcript text (may be empty for partials) |
|
||||
|
||||
### Semantics
|
||||
|
||||
- **`final: false`** — intermediate recognition result (partial). Robovoice
|
||||
logs these but does not act on them (only `final` triggers TTS).
|
||||
- **`final: true`** — complete utterance. Robovoice feeds this to the TTS
|
||||
engine and speaks it.
|
||||
|
||||
Malformed JSON or unknown field values are silently dropped by the client.
|
||||
Reference in New Issue
Block a user