Compare commits

...

36 Commits

Author SHA1 Message Date
mute a80bd8f669 rvsttd: increase fade-out to 25ms 2026-08-17 07:41:08 +00:00
mute c174576e14 rvsttd: increase fade-out to 10ms 2026-08-17 07:00:35 +00:00
mute b8ee014cc3 rvsttd: hold-back last drain chunk for fade, same faded audio to Moonshine + WAV 2026-08-17 06:58:18 +00:00
mute cc9fc95519 rvsttd: single faded audio stream feeds both Moonshine and debug WAV 2026-08-17 06:51:27 +00:00
mute 411f1c0952 rvsttd: fade-out on last audio chunk fed to Moonshine, debug WAV stays raw 2026-08-17 06:49:21 +00:00
mute b40a06b01b rvsttd: 5ms fade-out on debug WAV to eliminate trailing click 2026-08-17 06:47:46 +00:00
mute b6361ec080 Revert "fix click: 5ms fade-out from last sample to zero before silence padding"
This reverts commit 84474731e7.
2026-08-17 06:46:43 +00:00
mute 84474731e7 fix click: 5ms fade-out from last sample to zero before silence padding 2026-08-17 06:45:06 +00:00
mute b3e0fe3b30 fix: drain loop uses fixed 100ms deadline instead of idle timeout 2026-08-17 06:41:32 +00:00
mute 20e5c6c542 rvsttd: 100ms trailing audio drain before final flush, cpal stays alive during drain 2026-08-17 06:06:16 +00:00
mute 56bf55ba42 rvsttd: mpsc channel for audio capture, Fixed(800) buffer, deterministic shutdown 2026-08-17 05:57:01 +00:00
mute 004cd10f78 rvsttd: add --debug flag, save session audio (WAV) + transcript log 2026-08-14 06:39:58 +00:00
mute 8473ab6ba5 remove debug/trace logging and clean up log files 2026-08-13 12:51:41 +00:00
mute e09eb804b6 fix audio cutoff: don't Stop() after playback, let NAudio drain naturally 2026-08-13 12:50:20 +00:00
mute b4f05d6301 remove unused _recvThread field 2026-08-13 12:40:06 +00:00
mute caad0ed50e v0.9: VoicePipeline refactor — queue-based pipeline, lock/release model, thread-based TCP 2026-08-13 12:38:27 +00:00
mute 41a53d35b0 rvsttd: remove leftover session_id_counter references 2026-08-13 11:00:45 +00:00
mute fbbb9968b1 move PROTOCOL.md to repo root 2026-08-13 10:57:56 +00:00
mute b034ed15df rvsttd: enable TTS build (required for linking), fix unused var 2026-08-13 10:52:25 +00:00
mute b842f8b1f1 rvsttd: fix SocketAddr Default trait bound 2026-08-13 10:50:45 +00:00
mute d8bdec7543 rvsttd: drop ureq, use curl for model downloads (rustc 1.85 compat) 2026-08-13 10:49:49 +00:00
mute 23de208e96 rvsttd: fix build - strip -Werror, add TTS include path 2026-08-13 10:43:10 +00:00
mute 8811a15e1c rvsttd: patch all CMakeLists.txt for CMake 3.18 2026-08-13 10:37:43 +00:00
mute 912c380c02 rvsttd: patch CMake minimum version for Debian 11 2026-08-13 10:35:26 +00:00
mute b2b0c0d579 rvsttd: model fetch mode, build from source, ~/.rvsttd convention 2026-08-13 10:31:46 +00:00
mute 4be00bc4f1 add Rust target/ to .gitignore 2026-08-13 10:00:19 +00:00
mute 7765c6967c rename Server/server to rvsttd 2026-08-13 09:59:31 +00:00
mute facbfe6a5c v0.8: TCP transport via gatuna tunnel, session IDs, pre-synth playback, server Rust rewrite 2026-08-13 09:58:33 +00:00
mute ecf00c1bc4 v0.7: pre-synth buffering, session IDs, segment timeout playback 2026-08-12 10:30:11 +00:00
mute e1739059d9 v0.6: DHCP tunnel transport with NOP heartbeat protocol 2026-08-12 00:29:54 +00:00
mute 4a58b94f35 v0.5: TCP STT client with PTT on/off, server endpoint in config + UI 2026-08-11 09:03:50 +00:00
mute 08d6eeb46e v0.4: mic tray icon from mmres.dll, version in form title 2026-08-10 12:39:33 +00:00
mute fbbb52df90 v0.4: persistent config in AppData, voices in LOCALAPPDATA, remove Speak button 2026-08-10 12:32:07 +00:00
mute 9b0dc11ee6 v0.3: PTT hotkey capture, mouse support, model params
- Replace PTT key dropdown with live key-capture textbox
- Use low-level keyboard hook for capture (distinguishes Left/Right Ctrl)
- Support mouse buttons as PTT (Left/Right/Middle/X1/X2 via WH_MOUSE_LL)
- Filter mouse clicks from capture, allow Left/Right Ctrl
- Add Noise/Speed/NoiseW sliders with live labels, reinit on MouseUp
- Stream audio via BufferedWaveProvider (no temp WAV files)
- Add text input field with Speak button
- Fix sentence terminator, UTF-8 marshaling, voice catalogue JSON
2026-08-10 12:07:34 +00:00
mute f7ea07b78e v0.2: streaming playback, model params, text input
- Stream audio directly to device via BufferedWaveProvider (no temp WAVs)
- Add Noise/Speed/NoiseW sliders with live value labels
- Sliders reinit engine on MouseUp (not Scroll) to avoid locking
- Add text input field with Speak button for direct synthesis
- Test button uses text input if non-empty
- Fix sentence terminator auto-append (espeak-ng final-word drop)
- Fix UTF-8 text marshaling (piper_synthesize_start keeps text ref)
- Fix voice catalogue language column (JSON snake_case mapping)
- Use piper_default_synthesize_options for model-specific defaults
- Fix nullable warnings in designer files
- Fix unused _playing field in AudioOutput
2026-08-10 11:50:48 +00:00
mute 8585972a1e v0.1: Robovoice — PTT re-voicing app
WinForms tray app that captures text (from file or direct input) and
synthesizes speech via libpiper (Piper TTS), outputting to any audio
device (VB-CABLE for virtual mic routing).

Features:
- Global PTT hotkey (configurable F1-F12) via WH_KEYBOARD_LL
- Text file sequential reader (line-by-line on each PTT cycle)
- Direct text input with Speak button
- Voice manager: browse 147-voice Piper catalogue, download, remove
- libpiper P/Invoke wrapper with UTF-8 marshaling, streaming chunks
- BufferedWaveProvider streaming playback with trailing silence flush
- Sentence terminator auto-append (fixes espeak-ng final-word drop)
- Tray icon with minimize-to-tray

Architecture:
- Robovoice.Core: ITtsEngine, ISttSource interfaces, voice catalogue
- Robovoice.Tts.LibPiper: P/Invoke wrapper for libpiper.dll
- Robovoice.Stt.File: text file STT source (testing without mic server)
- Robovoice.Stt.Udp: UDP client stub (for future Linux mic server)
- Robovoice.App: WinForms UI, orchestrator, PTT hotkey, audio output
2026-08-10 11:27:28 +00:00
34 changed files with 3926 additions and 0 deletions
+32
View File
@@ -0,0 +1,32 @@
## .NET build output
bin/
obj/
## NuGet
*.nupkg
*.snupkg
.nuget/
## User-specific files
*.user
*.suo
.vs/
.idea/
## Test project (scratch)
Robovoice.Test/
## Native binaries (built from piper1-gpl, not source)
Robovoice.Tts.LibPiper/runtimes/
Robovoice.Tts.LibPiper/piper.h
## Voice models (downloaded at runtime)
*.onnx
*.onnx.json
## Temp files
*.wav
*.raw
## Rust build output
target/
+21
View File
@@ -0,0 +1,21 @@
# Robovoice Backlog
## Second-order PTT
When a game is running, it has its own PTT button (e.g. a voip push-to-talk
key). Robovoice should simulate pressing the game's PTT key before TTS audio
output begins, and release it after playback finishes.
This lets the TTS audio be transmitted through the game's voip channel to
other players.
### Considerations
- Needs a configurable "game PTT key" (separate from Robovoice's own PTT key)
- Use `SendInput` or `keybd_event` to synthesize the keypress
- Press the game PTT key right before buffered audio starts playing
- Release it after `AudioOutput` finishes playback (need a playback-complete
signal — currently `Flush()` doesn't provide one)
- Edge cases: what if the user presses Robovoice PTT while game PTT is still
held from a previous utterance? Flush should release game PTT too.
- Should this be a per-output-device setting? (CABLE Output vs speakers)
+106
View File
@@ -0,0 +1,106 @@
# Robovoice STT Protocol
## Overview
Robovoice connects to the STT server over TCP (typically through a gatuna
L2 tunnel). The server captures audio, runs Moonshine STT, and sends
transcript segments back. The client pre-synthesizes TTS on segments and
plays audio on final.
```
[Robovoice client] --TCP--> [STT server 127.0.0.1:6996]
│ │
├── ON <session>\n ────────►│ (abort old, start new session)
├── OFF <session>\n ────────►│ (stop, final STT pass)
│◄── P <session> <text>\n ──┤ (completed segment)
│◄── F <session> <text>\n ──┤ (all done; text may be empty)
```
## Transport
- **Protocol:** TCP (reliable, ordered, connection-oriented)
- **Server:** `127.0.0.1:6996` (hardcoded loopback)
- **Framing:** newline-delimited text (`\n`), UTF-8
- **Auto-reconnect:** client retries every 3s if connection drops
## Wire format
### Client → Server
**ON (PTT pressed):**
```
ON <session>\n
```
Starts a new STT session. The server aborts any active session and starts
recording. `<session>` is an incrementing unsigned integer chosen by the
client. Replies from the server echo this session ID.
**OFF (PTT released):**
```
OFF <session>\n
```
Stops the session. The server does a final STT pass on remaining audio and
sends any new segments followed by `F`.
### Server → Client
**Segment (completed VAD segment):**
```
P <session> <text>\n
```
A completed, VAD-separated utterance segment. The client starts TTS
synthesis immediately and buffers the audio (does not play yet).
**Final (all done):**
```
F <session> <text>\n
```
Signals all segments have been sent. `<text>` may be empty (`F <session>\n`).
Triggers playback of all buffered audio on the client. If text is non-empty,
the client synthesizes it before playing.
## Session IDs
- Client increments session ID on each PTT press
- Server echoes the session ID in all replies for that session
- Client drops any reply with a stale session ID (handles the race where
stale segments from an aborted session are still in the TCP buffer)
- Server aborts old session on receiving `ON` with a new session ID
## Client playback model
1. `P` arrives → start TTS synthesis immediately, buffer audio (don't play)
2. More `P` arrive → keep synthesizing and buffering
3. `F` arrives → play all buffered audio immediately
4. PTT pressed → flush: stop playback, cancel synthesis, clear buffers
The purpose of pre-synthesis is to minimize latency between PTT release
and audio playback. By the time `F` arrives, audio is already buffered.
## Server state machine
```
┌──────────┐ ON <session> ┌──────────────┐
│ IDLE │ ──────────────► │ RECORDING │
└──────────┘ └──────────────┘
│ │
OFF │ │
recv'd │ │
▼ │
┌─────────────┐
│ PROCESSING │
└─────────────┘
send │
P/F │
back to IDLE
```
- **IDLE → RECORDING:** `ON <session>` received, start mic capture
- **RECORDING:** Moonshine streaming produces completed segments → send `P`
- **RECORDING → PROCESSING:** `OFF <session>` received
- **PROCESSING → IDLE:** final STT pass, send remaining `P` + `F`
If `ON` arrives while recording, the current session is aborted (no final
flush) and a new session starts immediately.
+57
View File
@@ -0,0 +1,57 @@
using System.Text.Json;
using System.Windows.Forms;
namespace Robovoice.App;
public sealed class AppConfig
{
public string Voice { get; set; } = string.Empty;
public int PttKey { get; set; } = (int)Keys.F8;
public string OutputDevice { get; set; } = string.Empty;
public int NoiseScale { get; set; } = 667;
public int LengthScale { get; set; } = 100;
public int NoiseWScale { get; set; } = 800;
public bool MinimizeToTray { get; set; } = true;
public string SttEndpoint { get; set; } = "127.0.0.1:6996";
public static string AppDataDir => Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),
"Hackmaster", "Robovoice");
public static string ConfigPath => Path.Combine(AppDataDir, "config.json");
public static string VoicesDir => Path.Combine(AppDataDir, "Piper-voices");
private static readonly JsonSerializerOptions JsonOptions = new()
{
WriteIndented = true,
};
public static AppConfig Load()
{
try
{
if (File.Exists(ConfigPath))
{
using var stream = File.OpenRead(ConfigPath);
return JsonSerializer.Deserialize<AppConfig>(stream) ?? new AppConfig();
}
}
catch
{
}
return new AppConfig();
}
public void Save()
{
try
{
Directory.CreateDirectory(AppDataDir);
using var stream = File.Create(ConfigPath);
JsonSerializer.Serialize(stream, this, JsonOptions);
}
catch
{
}
}
}
+108
View File
@@ -0,0 +1,108 @@
using NAudio.Wave;
namespace Robovoice.App;
internal sealed class AudioOutput : IDisposable
{
private WaveOutEvent? _waveOut;
private BufferedWaveProvider? _bufferProvider;
private int _sampleRate;
private string _deviceName = string.Empty;
private bool _disposed;
public Action<string>? Log { get; set; }
public void Start(int sampleRate, string? deviceName = null)
{
ObjectDisposedException.ThrowIf(_disposed, this);
_sampleRate = sampleRate;
_deviceName = deviceName ?? string.Empty;
Stop();
int deviceNumber = FindDevice(_deviceName);
_waveOut = new WaveOutEvent { DeviceNumber = deviceNumber, DesiredLatency = 200 };
_bufferProvider = new BufferedWaveProvider(
WaveFormat.CreateIeeeFloatWaveFormat(sampleRate, 1))
{
BufferDuration = TimeSpan.FromSeconds(60),
DiscardOnBufferOverflow = true,
ReadFully = true,
};
_waveOut.Init(_bufferProvider);
_waveOut.PlaybackStopped += OnPlaybackStopped;
_waveOut.Play();
}
public void WriteSamples(float[] samples)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_bufferProvider is null) return;
byte[] bytes = new byte[samples.Length * sizeof(float)];
Buffer.BlockCopy(samples, 0, bytes, 0, bytes.Length);
_bufferProvider.AddSamples(bytes, 0, bytes.Length);
}
public void Flush()
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_bufferProvider is null || _sampleRate == 0) return;
int padSamples = (int)(_sampleRate * 0.5);
WriteSamples(new float[padSamples]);
}
private void OnPlaybackStopped(object? sender, StoppedEventArgs e)
{
}
public void Stop()
{
if (_waveOut is not null)
{
_waveOut.PlaybackStopped -= OnPlaybackStopped;
_waveOut.Stop();
_waveOut.Dispose();
_waveOut = null;
}
_bufferProvider?.ClearBuffer();
_bufferProvider = null;
}
private static int FindDevice(string? deviceName)
{
if (string.IsNullOrEmpty(deviceName))
return -1;
for (int i = 0; i < WaveOut.DeviceCount; i++)
{
var caps = WaveOut.GetCapabilities(i);
if (caps.ProductName.Contains(deviceName, StringComparison.OrdinalIgnoreCase))
return i;
}
return -1;
}
public static IReadOnlyList<(int Index, string Name)> GetDevices()
{
var devices = new List<(int, string)>();
for (int i = 0; i < WaveOut.DeviceCount; i++)
{
var caps = WaveOut.GetCapabilities(i);
devices.Add((i, caps.ProductName));
}
return devices;
}
public void Dispose()
{
if (_disposed) return;
Stop();
_disposed = true;
}
}
+56
View File
@@ -0,0 +1,56 @@
using System.Runtime.InteropServices;
namespace Robovoice.App;
internal static class IconExtractor
{
private const string Source = @"%SystemRoot%\System32\mmres.dll";
private static readonly int[] CandidateIndices = { 5, 12 };
public static Icon? TryExtractMicrophone(int size = 32)
{
string path = Environment.ExpandEnvironmentVariables(Source);
if (!File.Exists(path)) return null;
foreach (int index in CandidateIndices)
{
if (ExtractIconAt(path, index, size, size) is { } icon)
return icon;
}
return null;
}
private static Icon? ExtractIconAt(string path, int index, int width, int height)
{
IntPtr[] hicons = new IntPtr[1];
IntPtr[] ids = new IntPtr[1];
int count = PrivateExtractIcons(path, index, width, height, hicons, ids, 1, 0);
if (count <= 0 || hicons[0] == IntPtr.Zero) return null;
try
{
return Icon.FromHandle(hicons[0]);
}
catch
{
DestroyIcon(hicons[0]);
return null;
}
}
[DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern int PrivateExtractIcons(
string lpszFile,
int nIconIndex,
int cxIcon,
int cyIcon,
IntPtr[] phicon,
IntPtr[] phiconId,
int nIcons,
int flags);
[DllImport("user32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool DestroyIcon(IntPtr hIcon);
}
+239
View File
@@ -0,0 +1,239 @@
#nullable enable
namespace Robovoice.App;
partial class MainForm
{
private System.ComponentModel.IContainer? components = null;
private Label lblPttKey = null!;
private TextBox txtPttKey = null!;
private Label lblVoice = null!;
private ComboBox cmbVoice = null!;
private Button btnManageVoices = null!;
private Label lblOutput = null!;
private ComboBox cmbOutput = null!;
private Label lblServer = null!;
private TextBox txtSttEndpoint = null!;
private Button btnLock = null!;
private RichTextBox txtLog = null!;
private CheckBox chkMinimizeToTray = null!;
private Button btnClearLog = null!;
private Label lblNoise = null!;
private TrackBar trkNoise = null!;
private Label lblNoiseVal = null!;
private Label lblSpeed = null!;
private TrackBar trkSpeed = null!;
private Label lblSpeedVal = null!;
private Label lblNoiseW = null!;
private TrackBar trkNoiseW = null!;
private Label lblNoiseWVal = null!;
protected override void Dispose(bool disposing)
{
if (disposing && components != null)
components.Dispose();
base.Dispose(disposing);
}
private void InitializeComponent()
{
components = new System.ComponentModel.Container();
lblPttKey = new Label();
txtPttKey = new TextBox();
lblVoice = new Label();
cmbVoice = new ComboBox();
btnManageVoices = new Button();
lblOutput = new Label();
cmbOutput = new ComboBox();
lblServer = new Label();
txtSttEndpoint = new TextBox();
btnLock = new Button();
txtLog = new RichTextBox();
chkMinimizeToTray = new CheckBox();
btnClearLog = new Button();
lblNoise = new Label();
trkNoise = new TrackBar();
lblNoiseVal = new Label();
lblSpeed = new Label();
trkSpeed = new TrackBar();
lblSpeedVal = new Label();
lblNoiseW = new Label();
trkNoiseW = new TrackBar();
lblNoiseWVal = new Label();
SuspendLayout();
// lblPttKey
lblPttKey.Text = "PTT Key:";
lblPttKey.Location = new Point(12, 15);
lblPttKey.Size = new Size(60, 23);
lblPttKey.TextAlign = ContentAlignment.MiddleLeft;
// txtPttKey
txtPttKey.Location = new Point(75, 12);
txtPttKey.Size = new Size(80, 23);
txtPttKey.ReadOnly = true;
txtPttKey.TextAlign = HorizontalAlignment.Center;
txtPttKey.TabStop = false;
// lblVoice
lblVoice.Text = "Voice:";
lblVoice.Location = new Point(170, 15);
lblVoice.Size = new Size(45, 23);
lblVoice.TextAlign = ContentAlignment.MiddleLeft;
// cmbVoice
cmbVoice.Location = new Point(218, 12);
cmbVoice.Size = new Size(200, 23);
cmbVoice.DropDownStyle = ComboBoxStyle.DropDownList;
// btnManageVoices
btnManageVoices.Text = "Add/Remove...";
btnManageVoices.Location = new Point(423, 11);
btnManageVoices.Size = new Size(95, 25);
btnManageVoices.UseVisualStyleBackColor = true;
// lblOutput
lblOutput.Text = "Output:";
lblOutput.Location = new Point(530, 15);
lblOutput.Size = new Size(50, 23);
lblOutput.TextAlign = ContentAlignment.MiddleLeft;
// cmbOutput
cmbOutput.Location = new Point(583, 12);
cmbOutput.Size = new Size(150, 23);
cmbOutput.DropDownStyle = ComboBoxStyle.DropDownList;
// lblServer
lblServer.Text = "STT:";
lblServer.Location = new Point(12, 48);
lblServer.Size = new Size(35, 23);
lblServer.TextAlign = ContentAlignment.MiddleLeft;
// txtSttEndpoint
txtSttEndpoint.Location = new Point(50, 45);
txtSttEndpoint.Size = new Size(200, 23);
// btnLock
btnLock.Text = "Lock Model";
btnLock.Location = new Point(260, 44);
btnLock.Size = new Size(90, 25);
btnLock.UseVisualStyleBackColor = true;
// lblNoise
lblNoise.Text = "Noise:";
lblNoise.Location = new Point(12, 82);
lblNoise.Size = new Size(40, 23);
lblNoise.TextAlign = ContentAlignment.MiddleLeft;
// trkNoise
trkNoise.Location = new Point(52, 78);
trkNoise.Size = new Size(120, 45);
trkNoise.Minimum = 0;
trkNoise.Maximum = 1000;
trkNoise.Value = 667;
trkNoise.TickFrequency = 200;
trkNoise.Orientation = Orientation.Horizontal;
// lblNoiseVal
lblNoiseVal.Text = "0.667";
lblNoiseVal.Location = new Point(175, 82);
lblNoiseVal.Size = new Size(35, 23);
lblNoiseVal.TextAlign = ContentAlignment.MiddleLeft;
// lblSpeed
lblSpeed.Text = "Speed:";
lblSpeed.Location = new Point(220, 82);
lblSpeed.Size = new Size(40, 23);
lblSpeed.TextAlign = ContentAlignment.MiddleLeft;
// trkSpeed
trkSpeed.Location = new Point(260, 78);
trkSpeed.Size = new Size(120, 45);
trkSpeed.Minimum = 50;
trkSpeed.Maximum = 300;
trkSpeed.Value = 100;
trkSpeed.TickFrequency = 50;
trkSpeed.Orientation = Orientation.Horizontal;
// lblSpeedVal
lblSpeedVal.Text = "1.00";
lblSpeedVal.Location = new Point(383, 82);
lblSpeedVal.Size = new Size(35, 23);
lblSpeedVal.TextAlign = ContentAlignment.MiddleLeft;
// lblNoiseW
lblNoiseW.Text = "NoiseW:";
lblNoiseW.Location = new Point(428, 82);
lblNoiseW.Size = new Size(45, 23);
lblNoiseW.TextAlign = ContentAlignment.MiddleLeft;
// trkNoiseW
trkNoiseW.Location = new Point(475, 78);
trkNoiseW.Size = new Size(120, 45);
trkNoiseW.Minimum = 0;
trkNoiseW.Maximum = 1000;
trkNoiseW.Value = 800;
trkNoiseW.TickFrequency = 200;
trkNoiseW.Orientation = Orientation.Horizontal;
// lblNoiseWVal
lblNoiseWVal.Text = "0.800";
lblNoiseWVal.Location = new Point(598, 82);
lblNoiseWVal.Size = new Size(45, 23);
lblNoiseWVal.TextAlign = ContentAlignment.MiddleLeft;
// txtLog
txtLog.Location = new Point(12, 121);
txtLog.Size = new Size(800, 338);
txtLog.ReadOnly = true;
txtLog.Font = new Font("Consolas", 9F);
txtLog.BackColor = Color.FromArgb(30, 30, 30);
txtLog.ForeColor = Color.FromArgb(220, 220, 220);
// chkMinimizeToTray
chkMinimizeToTray.Text = "Minimize to tray on close";
chkMinimizeToTray.Location = new Point(12, 477);
chkMinimizeToTray.Size = new Size(180, 24);
chkMinimizeToTray.UseVisualStyleBackColor = true;
// btnClearLog
btnClearLog.Text = "Clear Log";
btnClearLog.Location = new Point(737, 475);
btnClearLog.Size = new Size(75, 25);
btnClearLog.UseVisualStyleBackColor = true;
// MainForm
AutoScaleDimensions = new SizeF(7F, 15F);
AutoScaleMode = AutoScaleMode.Font;
ClientSize = new Size(824, 509);
Controls.Add(lblPttKey);
Controls.Add(txtPttKey);
Controls.Add(lblVoice);
Controls.Add(cmbVoice);
Controls.Add(btnManageVoices);
Controls.Add(lblOutput);
Controls.Add(cmbOutput);
Controls.Add(lblServer);
Controls.Add(txtSttEndpoint);
Controls.Add(btnLock);
Controls.Add(lblNoise);
Controls.Add(trkNoise);
Controls.Add(lblNoiseVal);
Controls.Add(lblSpeed);
Controls.Add(trkSpeed);
Controls.Add(lblSpeedVal);
Controls.Add(lblNoiseW);
Controls.Add(trkNoiseW);
Controls.Add(lblNoiseWVal);
Controls.Add(txtLog);
Controls.Add(chkMinimizeToTray);
Controls.Add(btnClearLog);
MinimumSize = new Size(840, 547);
Text = "Robovoice";
FormBorderStyle = FormBorderStyle.FixedSingle;
MaximizeBox = false;
ResumeLayout(false);
}
}
+541
View File
@@ -0,0 +1,541 @@
using NAudio.Wave;
using Robovoice.App;
using Robovoice.Core;
using Robovoice.Core.Voices;
using Robovoice.Stt.Tcp;
using Robovoice.Tts.LibPiper;
using System.Diagnostics;
namespace Robovoice.App;
internal sealed partial class MainForm : Form
{
private readonly string _voicesDir = AppConfig.VoicesDir;
private readonly string _espeakDataPath;
private readonly AppConfig _config;
private LibPiperTtsEngine? _tts;
private AudioOutput? _audioOutput;
private TcpSttSource? _sttSource;
private VoicePipeline? _pipeline;
private PttHotkey? _pttHotkey;
private NotifyIcon? _trayIcon;
private bool _trayInit;
private bool _locked;
public MainForm()
{
InitializeComponent();
_espeakDataPath = Path.Combine(AppContext.BaseDirectory, "espeak-ng-data");
_config = AppConfig.Load();
Directory.CreateDirectory(AppConfig.AppDataDir);
Directory.CreateDirectory(_voicesDir);
Text = $"Robovoice {AppVersion}";
Load += OnLoad;
FormClosing += OnFormClosing;
}
private async void OnLoad(object? sender, EventArgs e)
{
try
{
ActiveControl = txtLog;
PopulateVoices();
PopulateOutputDevices();
_pttKey = (Keys)_config.PttKey;
if (_pttKey == Keys.None)
_pttKey = Keys.F8;
txtPttKey.Text = KeyToDisplayString(_pttKey);
if (!string.IsNullOrEmpty(_config.Voice) && cmbVoice.Items.Contains(_config.Voice))
cmbVoice.SelectedItem = _config.Voice;
else if (cmbVoice.Items.Count > 0)
cmbVoice.SelectedIndex = 0;
if (!string.IsNullOrEmpty(_config.OutputDevice) && cmbOutput.Items.Contains(_config.OutputDevice))
cmbOutput.SelectedItem = _config.OutputDevice;
else
AutoSelectCableOutput();
trkNoise.Value = _config.NoiseScale;
trkSpeed.Value = _config.LengthScale;
trkNoiseW.Value = _config.NoiseWScale;
OnSliderScroll(null, EventArgs.Empty);
chkMinimizeToTray.Checked = _config.MinimizeToTray;
txtSttEndpoint.Text = _config.SttEndpoint;
btnManageVoices.Click += OnManageVoices;
btnClearLog.Click += (_, _) => txtLog.Clear();
btnLock.Click += OnLockToggle;
txtPttKey.Enter += OnPttKeyFocus;
txtPttKey.KeyDown += OnPttKeyDown;
txtSttEndpoint.Leave += OnSttEndpointChanged;
trkNoise.Scroll += OnSliderScroll;
trkSpeed.Scroll += OnSliderScroll;
trkNoiseW.Scroll += OnSliderScroll;
trkNoise.MouseUp += OnSliderReleased;
trkSpeed.MouseUp += OnSliderReleased;
trkNoiseW.MouseUp += OnSliderReleased;
chkMinimizeToTray.CheckedChanged += (_, _) => SaveConfig();
Resize += OnResize;
SetupTray();
}
catch (Exception ex)
{
Debug.WriteLine($"OnLoad failed: {ex}");
}
}
private Keys _pttKey = Keys.F8;
private bool _capturingPttKey;
private PttHotkey? _captureHook;
private void OnPttKeyFocus(object? sender, EventArgs e)
{
_capturingPttKey = true;
txtPttKey.Text = "Press a key...";
txtPttKey.BackColor = Color.LightYellow;
_captureHook?.Dispose();
_captureHook = new PttHotkey();
_captureHook.CaptureKeyPressed += OnCaptureKey;
_captureHook.InstallCaptureHook();
}
private void OnCaptureKey(Keys key)
{
if (!_capturingPttKey) return;
if (key == Keys.Escape)
{
CancelCapture();
return;
}
if (key is Keys.ShiftKey or Keys.Menu or Keys.LWin or Keys.RWin)
return;
_pttKey = key;
_capturingPttKey = false;
txtPttKey.Text = KeyToDisplayString(_pttKey);
txtPttKey.BackColor = SystemColors.Window;
_captureHook?.Dispose();
_captureHook = null;
SetupHotkey();
SaveConfig();
}
private void CancelCapture()
{
_capturingPttKey = false;
txtPttKey.Text = KeyToDisplayString(_pttKey);
txtPttKey.BackColor = SystemColors.Window;
_captureHook?.Dispose();
_captureHook = null;
}
private void OnPttKeyDown(object? sender, KeyEventArgs e)
{
if (_capturingPttKey && e.KeyCode == Keys.Escape)
{
CancelCapture();
e.SuppressKeyPress = true;
}
}
private static string KeyToDisplayString(Keys key)
{
return key switch
{
>= Keys.F1 and <= Keys.F12 => key.ToString(),
Keys.RControlKey => "Right Ctrl",
Keys.LControlKey => "Left Ctrl",
Keys.Space => "Space",
Keys.LButton => "Mouse Left",
Keys.RButton => "Mouse Right",
Keys.MButton => "Mouse Middle",
Keys.XButton1 => "Mouse X1",
Keys.XButton2 => "Mouse X2",
Keys.Oemtilde => "`",
Keys.CapsLock => "CapsLock",
Keys.NumLock => "NumLock",
Keys.Scroll => "ScrollLock",
Keys.Pause => "Pause",
Keys.Insert => "Insert",
Keys.Delete => "Delete",
Keys.Home => "Home",
Keys.End => "End",
Keys.PageUp => "PageUp",
Keys.PageDown => "PageDown",
_ => key.ToString(),
};
}
private void PopulateVoices()
{
cmbVoice.Items.Clear();
if (!Directory.Exists(_voicesDir)) return;
foreach (var onnx in Directory.GetFiles(_voicesDir, "*.onnx"))
{
string name = Path.GetFileNameWithoutExtension(onnx);
cmbVoice.Items.Add(name);
}
}
private void PopulateOutputDevices()
{
cmbOutput.Items.Clear();
foreach (var (index, name) in AudioOutput.GetDevices())
{
cmbOutput.Items.Add(name);
}
}
private void AutoSelectCableOutput()
{
for (int i = 0; i < cmbOutput.Items.Count; i++)
{
if (cmbOutput.Items[i] is string s && s.Contains("CABLE", StringComparison.OrdinalIgnoreCase))
{
cmbOutput.SelectedIndex = i;
return;
}
}
if (cmbOutput.Items.Count > 0)
cmbOutput.SelectedIndex = 0;
}
// ─── Lock / Release ────────────────────────────────────────────────────
private async void OnLockToggle(object? sender, EventArgs e)
{
if (_locked)
{
ReleaseModel();
}
else
{
await LockModelAsync();
}
}
private async Task LockModelAsync()
{
if (cmbVoice.SelectedItem is not string voiceName)
{
Log("No voice selected.");
return;
}
string modelPath = Path.Combine(_voicesDir, voiceName + ".onnx");
if (!File.Exists(modelPath))
{
Log($"Model not found: {modelPath}");
return;
}
if (!Directory.Exists(_espeakDataPath))
{
Log($"espeak-ng-data not found at: {_espeakDataPath}");
return;
}
btnLock.Enabled = false;
Log($"Loading voice: {voiceName}...");
try
{
_tts = new LibPiperTtsEngine(
modelPath,
_espeakDataPath,
noiseScale: trkNoise.Value / 1000.0f,
lengthScale: trkSpeed.Value / 100.0f,
noiseWScale: trkNoiseW.Value / 1000.0f);
await Task.Run(() => _tts.InitializeAsync());
Log($"TTS ready (sample rate: {_tts.SampleRate} Hz)");
_audioOutput = new AudioOutput();
_sttSource = new TcpSttSource { Endpoint = txtSttEndpoint.Text, Log = Log };
_sttSource.TranscriptReceived += OnTranscript;
await _sttSource.StartAsync();
_pipeline = new VoicePipeline(_tts, _audioOutput, Log)
{
OutputDeviceName = cmbOutput.SelectedItem as string ?? string.Empty,
};
_pipeline.Start();
_locked = true;
SetControlsLocked(true);
btnLock.Text = "Release Model";
btnLock.Enabled = true;
SetupHotkey();
Log("Model locked. Press PTT to talk.");
}
catch (Exception ex)
{
Log($"Lock failed: {ex.Message}");
btnLock.Enabled = true;
_tts?.DisposeAsync().AsTask().Wait();
_tts = null;
_audioOutput?.Dispose();
_audioOutput = null;
if (_sttSource is not null)
{
_sttSource.TranscriptReceived -= OnTranscript;
_sttSource.DisposeAsync().AsTask().Wait();
_sttSource = null;
}
}
}
private void ReleaseModel()
{
if (!_locked) return;
_pttHotkey?.Dispose();
_pttHotkey = null;
if (_sttSource is not null)
{
_sttSource.TranscriptReceived -= OnTranscript;
_sttSource.DisposeAsync().AsTask().Wait();
_sttSource = null;
}
_pipeline?.Dispose();
_pipeline = null;
// VoicePipeline.Dispose calls _tts.DisposeAsync and _audioOutput.Dispose
_tts = null;
_audioOutput = null;
_locked = false;
SetControlsLocked(false);
btnLock.Text = "Lock Model";
Log("Model released.");
}
private void SetControlsLocked(bool locked)
{
cmbVoice.Enabled = !locked;
cmbOutput.Enabled = !locked;
trkNoise.Enabled = !locked;
trkSpeed.Enabled = !locked;
trkNoiseW.Enabled = !locked;
txtSttEndpoint.Enabled = !locked;
btnManageVoices.Enabled = !locked;
}
// ─── PTT ────────────────────────────────────────────────────────────────
private void SetupHotkey()
{
_pttHotkey?.Dispose();
_pttHotkey = new PttHotkey { Key = _pttKey };
_pttHotkey.Pressed += OnPttPressed;
_pttHotkey.Released += OnPttReleased;
_pttHotkey.Install();
Log($"PTT hotkey installed: {_pttHotkey.Key}");
}
private void OnPttPressed(object? sender, EventArgs e)
{
_pipeline?.OnPttPressed();
_sttSource?.SendOn();
}
private void OnPttReleased(object? sender, EventArgs e)
{
_sttSource?.SendOff();
_pipeline?.OnPttReleased();
}
// ─── STT transcript → pipeline ──────────────────────────────────────────
private void OnTranscript(object? sender, TranscriptEventArgs e)
{
if (_pipeline is null || !_locked) return;
var msg = e.Message;
if (msg.Type == TranscriptType.Partial)
{
if (!string.IsNullOrWhiteSpace(msg.Text))
{
Log($"SEGMENT: \"{msg.Text}\"");
_pipeline.EnqueueSegment(msg.Text);
}
}
else
{
if (!string.IsNullOrWhiteSpace(msg.Text))
{
Log($"FINAL: \"{msg.Text}\"");
_pipeline.EnqueueFinal(msg.Text);
}
else
{
Log("FINAL: (empty)");
_pipeline.EnqueueFinal("");
}
}
}
// ─── Voice manager ─────────────────────────────────────────────────────
private void OnManageVoices(object? sender, EventArgs e)
{
if (_locked)
{
Log("Release the model before managing voices.");
return;
}
string currentVoice = cmbVoice.SelectedItem as string ?? string.Empty;
using var dlg = new VoiceManagerForm(_voicesDir, currentVoice);
dlg.ShowDialog(this);
PopulateVoices();
if (!string.IsNullOrEmpty(dlg.SelectedVoiceKey) &&
cmbVoice.Items.Contains(dlg.SelectedVoiceKey))
{
cmbVoice.SelectedItem = dlg.SelectedVoiceKey;
}
else if (cmbVoice.Items.Count > 0)
{
cmbVoice.SelectedIndex = 0;
}
SaveConfig();
}
// ─── Slider / endpoint ─────────────────────────────────────────────────
private void OnSliderScroll(object? sender, EventArgs e)
{
lblNoiseVal.Text = $"{trkNoise.Value / 1000.0:F3}";
lblSpeedVal.Text = $"{trkSpeed.Value / 100.0:F2}";
lblNoiseWVal.Text = $"{trkNoiseW.Value / 1000.0:F3}";
}
private void OnSliderReleased(object? sender, MouseEventArgs e)
{
SaveConfig();
}
private void OnSttEndpointChanged(object? sender, EventArgs e)
{
_config.SttEndpoint = txtSttEndpoint.Text;
SaveConfig();
}
// ─── Tray ──────────────────────────────────────────────────────────────
private void SetupTray()
{
if (_trayInit) return;
_trayInit = true;
var micIcon = IconExtractor.TryExtractMicrophone();
if (micIcon is not null)
{
_trayIcon = new NotifyIcon
{
Icon = micIcon,
Text = $"Robovoice {AppVersion}",
Visible = true,
};
Icon = micIcon;
}
else
{
_trayIcon = new NotifyIcon
{
Icon = SystemIcons.Application,
Text = $"Robovoice {AppVersion}",
Visible = true,
};
}
var menu = new ContextMenuStrip();
menu.Items.Add("Show", null, (_, _) => ShowWindow());
menu.Items.Add("Exit", null, (_, _) =>
{
_trayIcon.Visible = false;
Application.Exit();
});
_trayIcon.ContextMenuStrip = menu;
_trayIcon.DoubleClick += (_, _) => ShowWindow();
}
private static string AppVersion =>
typeof(MainForm).Assembly.GetName().Version?.ToString() ?? "0.0";
private void OnResize(object? sender, EventArgs e)
{
if (WindowState == FormWindowState.Minimized && chkMinimizeToTray.Checked)
{
Hide();
WindowState = FormWindowState.Normal;
}
}
private void ShowWindow()
{
Show();
WindowState = FormWindowState.Normal;
Activate();
}
// ─── Config ────────────────────────────────────────────────────────────
private void SaveConfig()
{
_config.Voice = cmbVoice.SelectedItem as string ?? string.Empty;
_config.PttKey = (int)_pttKey;
_config.OutputDevice = cmbOutput.SelectedItem as string ?? string.Empty;
_config.NoiseScale = trkNoise.Value;
_config.LengthScale = trkSpeed.Value;
_config.NoiseWScale = trkNoiseW.Value;
_config.MinimizeToTray = chkMinimizeToTray.Checked;
_config.SttEndpoint = txtSttEndpoint.Text;
_config.Save();
}
// ─── Logging ───────────────────────────────────────────────────────────
private void Log(string message)
{
if (IsDisposed) return;
if (InvokeRequired)
{
BeginInvoke(() => Log(message));
return;
}
string timestamp = DateTime.Now.ToString("HH:mm:ss");
txtLog.AppendText($"[{timestamp}] {message}\n");
txtLog.ScrollToCaret();
}
// ─── Shutdown ──────────────────────────────────────────────────────────
private void OnFormClosing(object? sender, FormClosingEventArgs e)
{
_pttHotkey?.Dispose();
_trayIcon!.Visible = false;
ReleaseModel();
SaveConfig();
}
}
+11
View File
@@ -0,0 +1,11 @@
namespace Robovoice.App;
static class Program
{
[STAThread]
static void Main()
{
ApplicationConfiguration.Initialize();
Application.Run(new MainForm());
}
}
+186
View File
@@ -0,0 +1,186 @@
using System.Runtime.InteropServices;
using System.Windows.Forms;
namespace Robovoice.App;
internal sealed class PttHotkey : IDisposable
{
private const int WH_KEYBOARD_LL = 13;
private const int WH_MOUSE_LL = 14;
private const int WM_KEYDOWN = 0x0100;
private const int WM_KEYUP = 0x0101;
private const int WM_SYSKEYDOWN = 0x0104;
private const int WM_SYSKEYUP = 0x0105;
private const int WM_LBUTTONDOWN = 0x0201;
private const int WM_LBUTTONUP = 0x0202;
private const int WM_RBUTTONDOWN = 0x0204;
private const int WM_RBUTTONUP = 0x0205;
private const int WM_MBUTTONDOWN = 0x0207;
private const int WM_MBUTTONUP = 0x0208;
private const int WM_XBUTTONDOWN = 0x020B;
private const int WM_XBUTTONUP = 0x020C;
private readonly LowLevelKeyboardProc _kbProc;
private readonly LowLevelMouseProc _msProc;
private IntPtr _kbHook = IntPtr.Zero;
private IntPtr _msHook = IntPtr.Zero;
private bool _isDown;
private bool _disposed;
public Keys Key { get; set; } = Keys.F8;
public static bool IsMouseButton(Keys key) =>
key is Keys.LButton or Keys.RButton or Keys.MButton or Keys.XButton1 or Keys.XButton2;
public event EventHandler? Pressed;
public event EventHandler? Released;
public event Action<Keys>? CaptureKeyPressed;
public void InstallCaptureHook()
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_kbHook != IntPtr.Zero) return;
IntPtr hModule = GetModuleHandle(null);
_kbHook = SetWindowsHookEx(WH_KEYBOARD_LL, _kbProc, hModule, 0);
}
public PttHotkey()
{
_kbProc = KeyboardCallback;
_msProc = MouseCallback;
}
public void Install()
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_kbHook != IntPtr.Zero || _msHook != IntPtr.Zero) return;
IntPtr hModule = GetModuleHandle(null);
if (IsMouseButton(Key))
_msHook = SetWindowsHookEx(WH_MOUSE_LL, _msProc, hModule, 0);
else
_kbHook = SetWindowsHookEx(WH_KEYBOARD_LL, _kbProc, hModule, 0);
}
public void Uninstall()
{
if (_kbHook != IntPtr.Zero)
{
UnhookWindowsHookEx(_kbHook);
_kbHook = IntPtr.Zero;
}
if (_msHook != IntPtr.Zero)
{
UnhookWindowsHookEx(_msHook);
_msHook = IntPtr.Zero;
}
_isDown = false;
}
private IntPtr KeyboardCallback(int nCode, IntPtr wParam, IntPtr lParam)
{
if (nCode >= 0)
{
int vkCode = Marshal.ReadInt32(lParam);
Keys key = (Keys)vkCode;
bool isDown = wParam == WM_KEYDOWN || wParam == WM_SYSKEYDOWN;
bool isUp = wParam == WM_KEYUP || wParam == WM_SYSKEYUP;
if (CaptureKeyPressed is not null && isDown)
{
CaptureKeyPressed.Invoke(key);
}
else if (key == Key)
{
if (isDown && !_isDown)
{
_isDown = true;
Pressed?.Invoke(this, EventArgs.Empty);
}
else if (isUp && _isDown)
{
_isDown = false;
Released?.Invoke(this, EventArgs.Empty);
}
}
}
return CallNextHookEx(_kbHook, nCode, wParam, lParam);
}
private IntPtr MouseCallback(int nCode, IntPtr wParam, IntPtr lParam)
{
if (nCode >= 0)
{
Keys? button = null;
switch ((int)wParam)
{
case WM_LBUTTONDOWN: button = Keys.LButton; break;
case WM_LBUTTONUP: button = Keys.LButton; break;
case WM_RBUTTONDOWN: button = Keys.RButton; break;
case WM_RBUTTONUP: button = Keys.RButton; break;
case WM_MBUTTONDOWN: button = Keys.MButton; break;
case WM_MBUTTONUP: button = Keys.MButton; break;
case WM_XBUTTONDOWN:
case WM_XBUTTONUP:
{
int xButton = Marshal.ReadInt32(lParam + 8) >> 16;
button = xButton == 1 ? Keys.XButton1 : Keys.XButton2;
break;
}
}
if (button is Keys btn && btn == Key)
{
bool isDown = wParam == WM_LBUTTONDOWN || wParam == WM_RBUTTONDOWN
|| wParam == WM_MBUTTONDOWN || wParam == WM_XBUTTONDOWN;
bool isUp = wParam == WM_LBUTTONUP || wParam == WM_RBUTTONUP
|| wParam == WM_MBUTTONUP || wParam == WM_XBUTTONUP;
if (isDown && !_isDown)
{
_isDown = true;
Pressed?.Invoke(this, EventArgs.Empty);
}
else if (isUp && _isDown)
{
_isDown = false;
Released?.Invoke(this, EventArgs.Empty);
}
}
}
return CallNextHookEx(_msHook, nCode, wParam, lParam);
}
public void Dispose()
{
if (_disposed) return;
Uninstall();
_disposed = true;
}
private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);
private delegate IntPtr LowLevelMouseProc(int nCode, IntPtr wParam, IntPtr lParam);
[DllImport("user32.dll", SetLastError = true)]
private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);
[DllImport("user32.dll", SetLastError = true)]
private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelMouseProc lpfn, IntPtr hMod, uint dwThreadId);
[DllImport("user32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool UnhookWindowsHookEx(IntPtr hhk);
[DllImport("user32.dll", SetLastError = true)]
private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern IntPtr GetModuleHandle(string? lpModuleName);
}
+22
View File
@@ -0,0 +1,22 @@
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="..\Robovoice.Core\Robovoice.Core.csproj" />
<ProjectReference Include="..\Robovoice.Tts.LibPiper\Robovoice.Tts.LibPiper.csproj" />
<ProjectReference Include="..\Robovoice.Stt.Tcp\Robovoice.Stt.Tcp.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="NAudio" Version="2.3.0" />
</ItemGroup>
<PropertyGroup>
<OutputType>WinExe</OutputType>
<TargetFramework>net10.0-windows</TargetFramework>
<Nullable>enable</Nullable>
<UseWindowsForms>true</UseWindowsForms>
<ImplicitUsings>enable</ImplicitUsings>
<Version>0.4.0</Version>
</PropertyGroup>
</Project>
+117
View File
@@ -0,0 +1,117 @@
#nullable enable
namespace Robovoice.App;
partial class VoiceManagerForm
{
private System.ComponentModel.IContainer? components = null;
private Label lblFilter = null!;
private TextBox txtFilter = null!;
private DataGridView gridVoices = null!;
private Button btnDownload = null!;
private Button btnRemove = null!;
private Button btnClose = null!;
private Label lblStatus = null!;
private ProgressBar progressBar = null!;
protected override void Dispose(bool disposing)
{
if (disposing && components != null)
components.Dispose();
base.Dispose(disposing);
}
private void InitializeComponent()
{
components = new System.ComponentModel.Container();
lblFilter = new Label();
txtFilter = new TextBox();
gridVoices = new DataGridView();
btnDownload = new Button();
btnRemove = new Button();
btnClose = new Button();
lblStatus = new Label();
progressBar = new ProgressBar();
((System.ComponentModel.ISupportInitialize)gridVoices).BeginInit();
SuspendLayout();
// lblFilter
lblFilter.Text = "Filter:";
lblFilter.Location = new Point(12, 15);
lblFilter.Size = new Size(45, 23);
lblFilter.TextAlign = ContentAlignment.MiddleLeft;
// txtFilter
txtFilter.Location = new Point(60, 12);
txtFilter.Size = new Size(300, 23);
// gridVoices
gridVoices.Location = new Point(12, 42);
gridVoices.Size = new Size(660, 380);
gridVoices.AllowUserToAddRows = false;
gridVoices.AllowUserToDeleteRows = false;
gridVoices.AllowUserToResizeRows = false;
gridVoices.ReadOnly = true;
gridVoices.SelectionMode = DataGridViewSelectionMode.FullRowSelect;
gridVoices.MultiSelect = false;
gridVoices.RowHeadersVisible = false;
gridVoices.BackgroundColor = SystemColors.Window;
gridVoices.BorderStyle = BorderStyle.FixedSingle;
gridVoices.CellBorderStyle = DataGridViewCellBorderStyle.SingleHorizontal;
gridVoices.ColumnHeadersHeightSizeMode = DataGridViewColumnHeadersHeightSizeMode.AutoSize;
gridVoices.AutoSizeColumnsMode = DataGridViewAutoSizeColumnsMode.Fill;
// btnDownload
btnDownload.Text = "Download";
btnDownload.Location = new Point(12, 430);
btnDownload.Size = new Size(90, 28);
btnDownload.UseVisualStyleBackColor = true;
// btnRemove
btnRemove.Text = "Remove";
btnRemove.Location = new Point(108, 430);
btnRemove.Size = new Size(90, 28);
btnRemove.UseVisualStyleBackColor = true;
// btnClose
btnClose.Text = "Close";
btnClose.Location = new Point(582, 430);
btnClose.Size = new Size(90, 28);
btnClose.UseVisualStyleBackColor = true;
// lblStatus
lblStatus.Text = "";
lblStatus.Location = new Point(12, 465);
lblStatus.Size = new Size(660, 20);
lblStatus.TextAlign = ContentAlignment.MiddleLeft;
lblStatus.ForeColor = Color.DarkBlue;
// progressBar
progressBar.Location = new Point(208, 433);
progressBar.Size = new Size(360, 23);
progressBar.Visible = false;
// VoiceManagerForm
AutoScaleDimensions = new SizeF(7F, 15F);
AutoScaleMode = AutoScaleMode.Font;
ClientSize = new Size(684, 495);
Controls.Add(lblFilter);
Controls.Add(txtFilter);
Controls.Add(gridVoices);
Controls.Add(btnDownload);
Controls.Add(btnRemove);
Controls.Add(btnClose);
Controls.Add(lblStatus);
Controls.Add(progressBar);
MinimumSize = new Size(700, 535);
Text = "Voice Manager";
FormBorderStyle = FormBorderStyle.FixedSingle;
MaximizeBox = false;
StartPosition = FormStartPosition.CenterParent;
((System.ComponentModel.ISupportInitialize)gridVoices).EndInit();
ResumeLayout(false);
}
}
+285
View File
@@ -0,0 +1,285 @@
using Robovoice.Core.Voices;
using System.ComponentModel;
namespace Robovoice.App;
internal sealed partial class VoiceManagerForm : Form
{
private readonly string _voicesDir;
private readonly VoiceCatalogue _catalogue;
private IReadOnlyList<VoiceInfo> _allVoices = Array.Empty<VoiceInfo>();
private HashSet<string> _installedKeys = new();
private bool _catalogueLoaded;
public string? SelectedVoiceKey { get; private set; }
public VoiceManagerForm(string voicesDir, string? currentVoiceKey)
{
InitializeComponent();
_voicesDir = voicesDir;
_catalogue = new VoiceCatalogue();
SelectedVoiceKey = currentVoiceKey;
txtFilter.TextChanged += (_, _) => ApplyFilter();
btnDownload.Click += OnDownload;
btnRemove.Click += OnRemove;
btnClose.Click += (_, _) => Close();
gridVoices.SelectionChanged += OnSelectionChanged;
Load += OnLoad;
FormClosing += OnFormClosing;
}
private async void OnLoad(object? sender, EventArgs e)
{
btnDownload.Enabled = false;
btnRemove.Enabled = false;
RefreshInstalled();
lblStatus.Text = "Fetching voice catalogue...";
progressBar.Visible = true;
progressBar.Style = ProgressBarStyle.Marquee;
try
{
_allVoices = await _catalogue.GetCatalogueAsync();
_catalogueLoaded = true;
PopulateGrid();
lblStatus.Text = $"{_allVoices.Count} voices available, {_installedKeys.Count} installed";
}
catch (Exception ex)
{
lblStatus.Text = $"Failed to fetch catalogue: {ex.Message}";
}
finally
{
progressBar.Visible = false;
}
}
private void RefreshInstalled()
{
_installedKeys = VoiceCatalogue
.GetInstalledVoices(_voicesDir)
.ToHashSet();
}
private void PopulateGrid()
{
gridVoices.Rows.Clear();
gridVoices.Columns.Clear();
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Key",
HeaderText = "Key",
FillWeight = 40,
});
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Language",
HeaderText = "Language",
FillWeight = 25,
});
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Voice",
HeaderText = "Voice",
FillWeight = 20,
});
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Quality",
HeaderText = "Quality",
FillWeight = 15,
});
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Size",
HeaderText = "Size",
FillWeight = 15,
});
gridVoices.Columns.Add(new DataGridViewTextBoxColumn
{
Name = "Status",
HeaderText = "Status",
FillWeight = 15,
});
foreach (var voice in _allVoices)
{
bool installed = _installedKeys.Contains(voice.Key);
int rowIdx = gridVoices.Rows.Add(
voice.Key,
voice.Language.NameEnglish,
voice.Name,
voice.Quality,
voice.SizeDisplay,
installed ? "Installed" : "Available");
var row = gridVoices.Rows[rowIdx];
row.Tag = voice.Key;
if (installed)
{
row.DefaultCellStyle.BackColor = Color.FromArgb(235, 245, 235);
if (voice.Key == SelectedVoiceKey)
{
row.DefaultCellStyle.Font = new Font(gridVoices.Font, FontStyle.Bold);
row.Selected = true;
}
}
else
{
row.DefaultCellStyle.ForeColor = Color.Gray;
}
}
if (gridVoices.SelectedRows.Count == 0 && gridVoices.Rows.Count > 0)
gridVoices.Rows[0].Selected = true;
}
private void ApplyFilter()
{
string filter = txtFilter.Text.Trim().ToLowerInvariant();
if (string.IsNullOrEmpty(filter))
{
foreach (DataGridViewRow row in gridVoices.Rows)
row.Visible = true;
return;
}
foreach (DataGridViewRow row in gridVoices.Rows)
{
if (row.Tag is not string key)
{
row.Visible = false;
continue;
}
var voice = _allVoices.FirstOrDefault(v => v.Key == key);
if (voice is null)
{
row.Visible = false;
continue;
}
string haystack = $"{key} {voice.Name} {voice.Language.NameEnglish} {voice.Language.Code} {voice.Language.Family} {voice.Quality}".ToLowerInvariant();
row.Visible = haystack.Contains(filter);
}
}
private void OnSelectionChanged(object? sender, EventArgs e)
{
if (gridVoices.SelectedRows.Count == 0) return;
var row = gridVoices.SelectedRows[0];
if (row.Tag is not string key) return;
bool installed = _installedKeys.Contains(key);
btnDownload.Enabled = !installed && _catalogueLoaded;
btnRemove.Enabled = installed;
}
private async void OnDownload(object? sender, EventArgs e)
{
if (gridVoices.SelectedRows.Count == 0) return;
var row = gridVoices.SelectedRows[0];
if (row.Tag is not string key) return;
var voice = _allVoices.FirstOrDefault(v => v.Key == key);
if (voice is null) return;
btnDownload.Enabled = false;
btnRemove.Enabled = false;
progressBar.Visible = true;
progressBar.Style = ProgressBarStyle.Continuous;
progressBar.Value = 0;
lblStatus.Text = $"Downloading {key} ({voice.SizeDisplay})...";
try
{
var progress = new Progress<(long downloaded, long total)>(p =>
{
if (p.total > 0)
{
progressBar.Value = (int)(p.downloaded * 100 / p.total);
lblStatus.Text = $"Downloading {key}... {p.downloaded / (1024 * 1024)} / {p.total / (1024 * 1024)} MB";
}
});
await _catalogue.DownloadVoiceAsync(voice, _voicesDir, progress);
RefreshInstalled();
UpdateRowStatus(key, installed: true);
lblStatus.Text = $"Downloaded {key} successfully.";
}
catch (Exception ex)
{
lblStatus.Text = $"Download failed: {ex.Message}";
}
finally
{
progressBar.Visible = false;
OnSelectionChanged(null, EventArgs.Empty);
}
}
private void OnRemove(object? sender, EventArgs e)
{
if (gridVoices.SelectedRows.Count == 0) return;
var row = gridVoices.SelectedRows[0];
if (row.Tag is not string key) return;
var dlgResult = MessageBox.Show(
$"Remove voice '{key}'?\nThis will delete the .onnx and .onnx.json files.",
"Confirm Remove",
MessageBoxButtons.YesNo,
MessageBoxIcon.Question);
if (dlgResult != DialogResult.Yes) return;
try
{
VoiceCatalogue.RemoveVoice(_voicesDir, key);
RefreshInstalled();
UpdateRowStatus(key, installed: false);
if (SelectedVoiceKey == key)
SelectedVoiceKey = null;
lblStatus.Text = $"Removed {key}.";
}
catch (Exception ex)
{
lblStatus.Text = $"Remove failed: {ex.Message}";
}
OnSelectionChanged(null, EventArgs.Empty);
}
private void UpdateRowStatus(string key, bool installed)
{
foreach (DataGridViewRow row in gridVoices.Rows)
{
if (row.Tag is string rowKey && rowKey == key)
{
row.Cells["Status"].Value = installed ? "Installed" : "Available";
row.DefaultCellStyle.BackColor = installed
? Color.FromArgb(235, 245, 235)
: Color.White;
row.DefaultCellStyle.ForeColor = installed
? Color.Black
: Color.Gray;
break;
}
}
lblStatus.Text = $"{_allVoices.Count} voices available, {_installedKeys.Count} installed";
}
private void OnFormClosing(object? sender, FormClosingEventArgs e)
{
if (gridVoices.SelectedRows.Count > 0 && gridVoices.SelectedRows[0].Tag is string key)
{
if (_installedKeys.Contains(key))
SelectedVoiceKey = key;
}
_catalogue.DisposeAsync().AsTask().Wait();
}
}
+187
View File
@@ -0,0 +1,187 @@
using System.Collections.Concurrent;
using Robovoice.Core;
using Robovoice.Tts.LibPiper;
namespace Robovoice.App;
internal sealed record TextItem(uint Session, string Text);
internal sealed record AudioItem(uint Session, float[]? Samples);
internal sealed class VoicePipeline : IDisposable
{
private readonly LibPiperTtsEngine _tts;
private readonly AudioOutput _audioOutput;
private readonly Action<string> _log;
private BlockingCollection<TextItem> _textQueue = new();
private BlockingCollection<AudioItem> _audioQueue = new();
private ManualResetEventSlim _gate = new(false);
private volatile uint _currentSession;
private volatile bool _running;
private Thread? _synthThread;
private Thread? _playerThread;
private bool _disposed;
public string OutputDeviceName { get; set; } = string.Empty;
public VoicePipeline(
LibPiperTtsEngine tts,
AudioOutput audioOutput,
Action<string> log)
{
_tts = tts;
_audioOutput = audioOutput;
_audioOutput.Log = log;
_log = log;
}
public void Start()
{
_running = true;
_synthThread = new Thread(SynthLoop) { IsBackground = true, Name = "VoicePipeline-Synth" };
_playerThread = new Thread(PlayerLoop) { IsBackground = true, Name = "VoicePipeline-Player" };
_synthThread.Start();
_playerThread.Start();
}
public void EnqueueSegment(string text)
{
if (!_running) return;
uint session = _currentSession;
_textQueue.Add(new TextItem(session, text));
}
public void EnqueueFinal(string text)
{
if (!_running) return;
uint session = _currentSession;
if (!string.IsNullOrEmpty(text))
_textQueue.Add(new TextItem(session, text));
_textQueue.Add(new TextItem(session, ""));
_gate.Reset();
}
public void OnPttPressed()
{
_currentSession++;
_gate.Reset();
_audioOutput.Stop();
}
public void OnPttReleased()
{
_gate.Set();
}
private void SynthLoop()
{
foreach (var item in _textQueue.GetConsumingEnumerable())
{
if (!_running) break;
if (item.Session != _currentSession)
continue;
if (item.Text.Length == 0)
{
// :F sentinel — signal end of utterance to player
_audioQueue.Add(new AudioItem(item.Session, null));
continue;
}
try
{
foreach (var chunk in _tts.SynthesizeSync(item.Text))
{
if (item.Session != _currentSession)
{
// Session changed mid-synthesis — drain piper cleanly
_tts.SynthesizeDrain();
break;
}
_audioQueue.Add(new AudioItem(item.Session, chunk.Samples));
}
}
catch (Exception ex)
{
_log($"TTS error: {ex.Message}");
}
}
}
private void PlayerLoop()
{
while (_running)
{
_gate.Wait();
if (!_running) break;
// Drain stale items, then wait for real audio for current session
AudioItem firstItem = default!;
while (_running)
{
try { firstItem = _audioQueue.Take(); }
catch (InvalidOperationException) { return; }
if (firstItem.Session != _currentSession)
continue;
break;
}
if (!_running) break;
if (firstItem.Samples == null)
{
// :F with no audio — nothing to play
_gate.Reset();
continue;
}
_audioOutput.Start(_tts.SampleRate, OutputDeviceName);
_audioOutput.WriteSamples(firstItem.Samples);
_log($"TTS: playback started (session {firstItem.Session})");
foreach (var item in _audioQueue.GetConsumingEnumerable())
{
if (!_running) break;
if (item.Session != _currentSession)
break;
if (item.Samples == null)
break;
_audioOutput.WriteSamples(item.Samples);
}
_audioOutput.Flush();
_log("TTS: playback finished");
_gate.Reset();
}
}
public void Dispose()
{
if (_disposed) return;
_disposed = true;
_running = false;
_gate.Set();
_textQueue.CompleteAdding();
_audioQueue.CompleteAdding();
_synthThread?.Join(5000);
_playerThread?.Join(5000);
_textQueue.Dispose();
_audioQueue.Dispose();
_gate.Dispose();
_tts.DisposeAsync().AsTask().Wait();
_audioOutput.Dispose();
}
}
+3
View File
@@ -0,0 +1,3 @@
namespace Robovoice.Core;
public sealed record AudioChunk(float[] Samples, int SampleRate);
+17
View File
@@ -0,0 +1,17 @@
namespace Robovoice.Core;
public interface ISttSource : IAsyncDisposable
{
event TranscriptEventHandler? TranscriptReceived;
Task StartAsync(CancellationToken ct = default);
Task StopAsync(CancellationToken ct = default);
}
public sealed class TranscriptEventArgs : EventArgs
{
public TranscriptMessage Message { get; init; } = new(TranscriptType.Final, string.Empty);
}
public delegate void TranscriptEventHandler(object? sender, TranscriptEventArgs e);
+27
View File
@@ -0,0 +1,27 @@
using System.Runtime.CompilerServices;
namespace Robovoice.Core;
public interface ITtsEngine : IAsyncDisposable
{
string Name { get; }
int SampleRate { get; }
Task InitializeAsync(CancellationToken ct = default);
IAsyncEnumerable<AudioChunk> SynthesizeAsync(
string text,
CancellationToken ct = default);
}
public static class TtsEngineExtensions
{
public static ConfiguredCancelableAsyncEnumerable<AudioChunk> SynthesizeAsync(
this ITtsEngine engine,
string text,
CancellationToken ct = default)
{
return engine.SynthesizeAsync(text, ct).ConfigureAwait(false).WithCancellation(ct);
}
}
+9
View File
@@ -0,0 +1,9 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>
+9
View File
@@ -0,0 +1,9 @@
namespace Robovoice.Core;
public enum TranscriptType
{
Partial,
Final,
}
public sealed record TranscriptMessage(TranscriptType Type, string Text);
+131
View File
@@ -0,0 +1,131 @@
using System.Collections.Concurrent;
using System.Net.Http.Json;
using System.Security.Cryptography;
using System.Text.Json;
namespace Robovoice.Core.Voices;
public sealed class VoiceCatalogue : IAsyncDisposable
{
private const string CatalogueUrl =
"https://huggingface.co/rhasspy/piper-voices/resolve/main/voices.json";
private const string DownloadBaseUrl =
"https://huggingface.co/rhasspy/piper-voices/resolve/main/";
private readonly HttpClient _http;
private List<VoiceInfo>? _catalogue;
private bool _disposed;
public VoiceCatalogue()
{
_http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };
}
public async Task<IReadOnlyList<VoiceInfo>> GetCatalogueAsync(
CancellationToken ct = default)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_catalogue is not null)
return _catalogue;
var dict = await _http.GetFromJsonAsync<Dictionary<string, VoiceInfo>>(
CatalogueUrl, ct)
?? throw new InvalidOperationException("Failed to fetch voice catalogue.");
_catalogue = dict.Values
.OrderBy(v => v.Language.NameEnglish)
.ThenBy(v => v.Name)
.ThenBy(v => v.Quality)
.ToList();
return _catalogue;
}
public static IReadOnlyList<string> GetInstalledVoices(string voicesDir)
{
if (!Directory.Exists(voicesDir))
return Array.Empty<string>();
return Directory.GetFiles(voicesDir, "*.onnx")
.Select(f => Path.GetFileNameWithoutExtension(f)!)
.OrderBy(n => n)
.ToList();
}
public static bool IsVoiceInstalled(string voicesDir, string voiceKey)
{
string onnxPath = Path.Combine(voicesDir, voiceKey + ".onnx");
string jsonPath = Path.Combine(voicesDir, voiceKey + ".onnx.json");
return File.Exists(onnxPath) && File.Exists(jsonPath);
}
public async Task DownloadVoiceAsync(
VoiceInfo voice,
string voicesDir,
IProgress<(long downloaded, long total)>? progress = null,
CancellationToken ct = default)
{
ObjectDisposedException.ThrowIf(_disposed, this);
Directory.CreateDirectory(voicesDir);
var onnxFile = voice.Files.GetValueOrDefault(voice.OnnxFileKey)
?? throw new InvalidOperationException($"No .onnx file for {voice.Key}");
var jsonFile = voice.Files.GetValueOrDefault(voice.JsonFileKey)
?? throw new InvalidOperationException($"No .onnx.json file for {voice.Key}");
string onnxPath = Path.Combine(voicesDir, voice.Key + ".onnx");
string jsonPath = Path.Combine(voicesDir, voice.Key + ".onnx.json");
await DownloadFileAsync(
DownloadBaseUrl + voice.OnnxFileKey,
onnxPath, onnxFile.SizeBytes, onnxFile.Md5Digest, progress, ct);
await DownloadFileAsync(
DownloadBaseUrl + voice.JsonFileKey,
jsonPath, jsonFile.SizeBytes, jsonFile.Md5Digest, null, ct);
}
private async Task DownloadFileAsync(
string url,
string destPath,
long expectedSize,
string expectedMd5,
IProgress<(long, long)>? progress,
CancellationToken ct)
{
using var resp = await _http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, ct);
resp.EnsureSuccessStatusCode();
long total = resp.Content.Headers.ContentLength ?? expectedSize;
long downloaded = 0;
await using var contentStream = await resp.Content.ReadAsStreamAsync(ct);
await using var fileStream = File.Create(destPath);
byte[] buffer = new byte[81920];
int read;
while ((read = await contentStream.ReadAsync(buffer, ct)) > 0)
{
await fileStream.WriteAsync(buffer.AsMemory(0, read), ct);
downloaded += read;
progress?.Report((downloaded, total));
}
}
public static void RemoveVoice(string voicesDir, string voiceKey)
{
string onnxPath = Path.Combine(voicesDir, voiceKey + ".onnx");
string jsonPath = Path.Combine(voicesDir, voiceKey + ".onnx.json");
if (File.Exists(onnxPath)) File.Delete(onnxPath);
if (File.Exists(jsonPath)) File.Delete(jsonPath);
}
public ValueTask DisposeAsync()
{
if (_disposed) return ValueTask.CompletedTask;
_http.Dispose();
_disposed = true;
return ValueTask.CompletedTask;
}
}
+65
View File
@@ -0,0 +1,65 @@
using System.Text.Json.Serialization;
namespace Robovoice.Core.Voices;
public sealed class VoiceInfo
{
public string Key { get; set; } = string.Empty;
public string Name { get; set; } = string.Empty;
public VoiceLanguage Language { get; set; } = new();
public string Quality { get; set; } = string.Empty;
public int NumSpeakers { get; set; }
public VoiceFiles Files { get; set; } = new();
public List<string> Aliases { get; set; } = new();
[JsonIgnore]
public string DisplayName =>
$"{Language.NameEnglish} ({Language.Code}) — {Name} [{Quality}]";
[JsonIgnore]
public long SizeBytes =>
Files.FirstOrDefault(f => f.Key.EndsWith(".onnx")).Value?.SizeBytes ?? 0;
[JsonIgnore]
public string SizeDisplay
{
get
{
double mb = SizeBytes / (1024.0 * 1024.0);
return mb >= 1024 ? $"{mb / 1024:F1} GB" : $"{mb:F0} MB";
}
}
[JsonIgnore]
public string OnnxFileKey => Files.Keys.FirstOrDefault(k => k.EndsWith(".onnx")) ?? "";
[JsonIgnore]
public string JsonFileKey => Files.Keys.FirstOrDefault(k => k.EndsWith(".onnx.json")) ?? "";
}
public sealed class VoiceLanguage
{
public string Code { get; set; } = string.Empty;
public string Family { get; set; } = string.Empty;
public string Region { get; set; } = string.Empty;
[JsonPropertyName("name_native")]
public string NameNative { get; set; } = string.Empty;
[JsonPropertyName("name_english")]
public string NameEnglish { get; set; } = string.Empty;
[JsonPropertyName("country_english")]
public string CountryEnglish { get; set; } = string.Empty;
}
public sealed class VoiceFiles : Dictionary<string, VoiceFile> { }
public sealed class VoiceFile
{
[JsonPropertyName("size_bytes")]
public long SizeBytes { get; set; }
[JsonPropertyName("md5_digest")]
public string Md5Digest { get; set; } = string.Empty;
}
+83
View File
@@ -0,0 +1,83 @@
using Robovoice.Core;
namespace Robovoice.Stt.File;
public sealed class FileSttSource : ISttSource
{
private readonly object _lock = new();
private string[] _lines = Array.Empty<string>();
private int _currentIndex;
private bool _disposed;
public event TranscriptEventHandler? TranscriptReceived;
public string FilePath { get; set; } = string.Empty;
public int LineCount { get; private set; }
public int CurrentIndex => _currentIndex;
public void LoadFile(string path)
{
ObjectDisposedException.ThrowIf(_disposed, this);
FilePath = path;
_lines = System.IO.File.ReadAllLines(path);
LineCount = _lines.Length;
_currentIndex = 0;
}
public string? GetPendingLine()
{
lock (_lock)
{
if (_lines.Length == 0) return null;
int idx = _currentIndex % _lines.Length;
return _lines[idx];
}
}
public void EmitNext()
{
ObjectDisposedException.ThrowIf(_disposed, this);
string? line = null;
lock (_lock)
{
if (_lines.Length > 0)
{
int idx = _currentIndex % _lines.Length;
line = _lines[idx];
_currentIndex++;
}
}
if (!string.IsNullOrWhiteSpace(line))
{
TranscriptReceived?.Invoke(this, new TranscriptEventArgs
{
Message = new TranscriptMessage(TranscriptType.Final, line!),
});
}
}
public void Reset()
{
lock (_lock)
{
_currentIndex = 0;
}
}
public Task StartAsync(CancellationToken ct = default)
{
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken ct = default)
{
return Task.CompletedTask;
}
public ValueTask DisposeAsync()
{
_disposed = true;
return ValueTask.CompletedTask;
}
}
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="..\Robovoice.Core\Robovoice.Core.csproj" />
</ItemGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="..\Robovoice.Core\Robovoice.Core.csproj" />
</ItemGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
</Project>
+291
View File
@@ -0,0 +1,291 @@
using System.Net;
using System.Net.Sockets;
using System.Text;
using Robovoice.Core;
namespace Robovoice.Stt.Tcp;
public sealed class TcpSttSource : ISttSource
{
private TcpClient? _tcp;
private NetworkStream? _stream;
private StreamWriter? _writer;
private Thread? _connectThread;
private volatile bool _running;
private readonly object _sendLock = new();
private uint _session;
private bool _disposed;
public string Endpoint { get; set; } = "127.0.0.1:6996";
public Action<string>? Log { get; set; }
public event TranscriptEventHandler? TranscriptReceived;
public Task StartAsync(CancellationToken ct = default)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_running)
return Task.CompletedTask;
_running = true;
_connectThread = new Thread(ConnectLoop) { IsBackground = true, Name = "TcpSttSource-Connect" };
_connectThread.Start();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken ct = default)
{
_running = false;
lock (_sendLock)
{
_writer?.Dispose();
_stream?.Dispose();
_tcp?.Close();
_writer = null;
_stream = null;
_tcp = null;
}
// Threads are background — they'll die when the process exits.
// Closing the socket unblocks any pending Read.
_connectThread?.Join(1000);
return Task.CompletedTask;
}
private void ConnectLoop()
{
while (_running)
{
IPEndPoint? endpoint = ParseEndpoint(Endpoint);
if (endpoint is null)
{
Log?.Invoke($"STT: invalid endpoint '{Endpoint}'");
SleepInterruptible(3000);
continue;
}
try
{
var tcp = new TcpClient();
tcp.Connect(endpoint.Address, endpoint.Port);
tcp.NoDelay = true;
lock (_sendLock)
{
_tcp = tcp;
_stream = tcp.GetStream();
_writer = new StreamWriter(_stream, new UTF8Encoding(false)) { AutoFlush = true };
}
Log?.Invoke($"STT: connected to {Endpoint}");
// Blocking receive loop — runs until disconnected or stopped.
ReceiveLoop();
Log?.Invoke("STT: disconnected");
}
catch (Exception ex)
{
if (_running)
Log?.Invoke($"STT: connection failed ({ex.Message}), retrying...");
}
finally
{
lock (_sendLock)
{
_writer?.Dispose();
_stream?.Dispose();
_tcp?.Close();
_writer = null;
_stream = null;
_tcp = null;
}
}
if (_running)
SleepInterruptible(3000);
}
}
private void ReceiveLoop()
{
byte[] buffer = new byte[4096];
StringBuilder lineBuf = new();
while (_running)
{
NetworkStream? stream;
lock (_sendLock)
{
stream = _stream;
}
if (stream is null)
break;
int bytesRead;
try
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
}
catch
{
break;
}
if (bytesRead == 0)
break;
for (int i = 0; i < bytesRead; i++)
{
byte b = buffer[i];
if (b == '\n')
{
string line = lineBuf.ToString().TrimEnd('\r');
lineBuf.Clear();
TranscriptMessage? message = ParseReply(line);
if (message is not null)
{
TranscriptReceived?.Invoke(this, new TranscriptEventArgs
{
Message = message,
});
}
}
else
{
lineBuf.Append((char)b);
}
}
}
}
private void SleepInterruptible(int ms)
{
int slice = 100;
int waited = 0;
while (_running && waited < ms)
{
int chunk = Math.Min(slice, ms - waited);
Thread.Sleep(chunk);
waited += chunk;
}
}
public void SendOn()
{
_session++;
Send($"ON {_session}");
}
public void SendOff()
{
Send($"OFF {_session}");
}
private void Send(string message)
{
lock (_sendLock)
{
if (_writer is null)
return;
try
{
_writer.WriteLine(message);
}
catch
{
Log?.Invoke($"STT: failed to send '{message}' (not connected?)");
}
}
}
private static IPEndPoint? ParseEndpoint(string endpoint)
{
int colon = endpoint.LastIndexOf(':');
if (colon <= 0)
return null;
string host = endpoint[..colon];
if (!int.TryParse(endpoint[(colon + 1)..], out int port))
return null;
if (IPAddress.TryParse(host, out var addr))
return new IPEndPoint(addr, port);
try
{
var addresses = Dns.GetHostAddresses(host);
addr = addresses.FirstOrDefault(a => a.AddressFamily == AddressFamily.InterNetwork);
if (addr is null)
return null;
return new IPEndPoint(addr, port);
}
catch
{
return null;
}
}
private TranscriptMessage? ParseReply(string line)
{
if (line.StartsWith("P ", StringComparison.Ordinal))
{
string rest = line["P ".Length..];
int space = rest.IndexOf(' ');
if (space < 0)
return null;
if (!uint.TryParse(rest[..space], out uint session))
return null;
if (session != _session)
{
Log?.Invoke($"STT: dropping stale reply (session {session} != current {_session})");
return null;
}
return new TranscriptMessage(TranscriptType.Partial, rest[(space + 1)..]);
}
if (line.StartsWith("F ", StringComparison.Ordinal))
{
string rest = line["F ".Length..];
int space = rest.IndexOf(' ');
if (space < 0)
{
if (uint.TryParse(rest, out uint session) && session == _session)
return new TranscriptMessage(TranscriptType.Final, string.Empty);
return null;
}
if (!uint.TryParse(rest[..space], out uint ses))
return null;
if (ses != _session)
{
Log?.Invoke($"STT: dropping stale reply (session {ses} != current {_session})");
return null;
}
return new TranscriptMessage(TranscriptType.Final, rest[(space + 1)..]);
}
return null;
}
public ValueTask DisposeAsync()
{
if (_disposed) return ValueTask.CompletedTask;
_disposed = true;
StopAsync();
return ValueTask.CompletedTask;
}
}
+244
View File
@@ -0,0 +1,244 @@
using System.Runtime.InteropServices;
using Robovoice.Core;
namespace Robovoice.Tts.LibPiper;
public sealed class LibPiperTtsEngine : ITtsEngine
{
private readonly string _modelPath;
private readonly string _espeakDataPath;
private readonly float? _noiseScaleOverride;
private readonly float? _lengthScaleOverride;
private readonly float? _noiseWScaleOverride;
private float _noiseScale;
private float _lengthScale;
private float _noiseWScale;
private IntPtr _synth;
private int _sampleRate;
private bool _initialized;
private bool _disposed;
public string Name => "libpiper";
public int SampleRate => _sampleRate;
public LibPiperTtsEngine(
string modelPath,
string espeakDataPath,
float? noiseScale = null,
float? lengthScale = null,
float? noiseWScale = null)
{
_modelPath = modelPath;
_espeakDataPath = espeakDataPath;
_noiseScaleOverride = noiseScale;
_lengthScaleOverride = lengthScale;
_noiseWScaleOverride = noiseWScale;
_noiseScale = noiseScale ?? 0.667f;
_lengthScale = lengthScale ?? 1.0f;
_noiseWScale = noiseWScale ?? 0.8f;
}
public Task InitializeAsync(CancellationToken ct = default)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (_initialized)
return Task.CompletedTask;
NativeDependencyLoader.EnsureLoaded();
_synth = PiperCreateUtf8(_modelPath, _modelPath + ".json", _espeakDataPath);
if (_synth == IntPtr.Zero)
throw new InvalidOperationException(
$"piper_create failed for model: {_modelPath}");
var defaults = PiperNative.piper_default_synthesize_options(_synth);
_noiseScale = _noiseScaleOverride ?? defaults.NoiseScale;
_lengthScale = _lengthScaleOverride ?? defaults.LengthScale;
_noiseWScale = _noiseWScaleOverride ?? defaults.NoiseWScale;
_sampleRate = 22050;
_initialized = true;
return Task.CompletedTask;
}
public async IAsyncEnumerable<AudioChunk> SynthesizeAsync(
string text,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken ct = default)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (!_initialized)
throw new InvalidOperationException("Engine not initialized.");
var options = new PiperSynthesizeOptions
{
SpeakerId = 0,
LengthScale = _lengthScale,
NoiseScale = _noiseScale,
NoiseWScale = _noiseWScale,
};
byte[] textBytes = System.Text.Encoding.UTF8.GetBytes(EnsureTerminator(text) + "\0");
GCHandle textPin = GCHandle.Alloc(textBytes, GCHandleType.Pinned);
try
{
int startResult = PiperNative.piper_synthesize_start(
_synth, textPin.AddrOfPinnedObject(), in options);
if (startResult != PiperNative.PiperOk)
throw new InvalidOperationException($"piper_synthesize_start failed: {startResult}");
while (true)
{
ct.ThrowIfCancellationRequested();
PiperAudioChunk chunk = default;
int result = await Task.Run(() => PiperNative.piper_synthesize_next(_synth, out chunk), ct);
if (chunk.NumSamples > 0 && chunk.Samples != IntPtr.Zero)
{
int numSamples = (int)chunk.NumSamples;
float[] samples = new float[numSamples];
Marshal.Copy(chunk.Samples, samples, 0, numSamples);
if (chunk.SampleRate > 0)
_sampleRate = chunk.SampleRate;
yield return new AudioChunk(samples, _sampleRate);
}
if (result == PiperNative.PiperDone || chunk.IsLast)
break;
if (result < 0)
throw new InvalidOperationException($"piper_synthesize_next failed: {result}");
}
}
finally
{
textPin.Free();
}
}
public IEnumerable<AudioChunk> SynthesizeSync(string text)
{
ObjectDisposedException.ThrowIf(_disposed, this);
if (!_initialized)
throw new InvalidOperationException("Engine not initialized.");
var options = new PiperSynthesizeOptions
{
SpeakerId = 0,
LengthScale = _lengthScale,
NoiseScale = _noiseScale,
NoiseWScale = _noiseWScale,
};
byte[] textBytes = System.Text.Encoding.UTF8.GetBytes(EnsureTerminator(text) + "\0");
GCHandle textPin = GCHandle.Alloc(textBytes, GCHandleType.Pinned);
try
{
int startResult = PiperNative.piper_synthesize_start(
_synth, textPin.AddrOfPinnedObject(), in options);
if (startResult != PiperNative.PiperOk)
throw new InvalidOperationException($"piper_synthesize_start failed: {startResult}");
while (true)
{
PiperAudioChunk chunk = default;
int result = PiperNative.piper_synthesize_next(_synth, out chunk);
if (chunk.NumSamples > 0 && chunk.Samples != IntPtr.Zero)
{
int numSamples = (int)chunk.NumSamples;
float[] samples = new float[numSamples];
Marshal.Copy(chunk.Samples, samples, 0, numSamples);
if (chunk.SampleRate > 0)
_sampleRate = chunk.SampleRate;
yield return new AudioChunk(samples, _sampleRate);
}
if (result == PiperNative.PiperDone || chunk.IsLast)
break;
if (result < 0)
throw new InvalidOperationException($"piper_synthesize_next failed: {result}");
}
}
finally
{
textPin.Free();
}
}
public void SynthesizeDrain()
{
if (!_initialized || _synth == IntPtr.Zero)
return;
PiperAudioChunk chunk;
while (PiperNative.piper_synthesize_next(_synth, out chunk) != PiperNative.PiperDone)
{
if (chunk.IsLast) break;
}
}
private static string EnsureTerminator(string text)
{
string trimmed = text.TrimEnd();
if (trimmed.Length == 0)
return text;
char last = trimmed[^1];
if (last is '.' or '!' or '?' or ',' or ';' or ':' or ')' or ']' or '}' or '"' or '\'' or '。' or '' or '')
return text;
return trimmed + ".";
}
private static IntPtr PiperCreateUtf8(string modelPath, string? configPath, string espeakDataPath)
{
byte[] modelBytes = System.Text.Encoding.UTF8.GetBytes(modelPath + "\0");
byte[] espeakBytes = System.Text.Encoding.UTF8.GetBytes(espeakDataPath + "\0");
GCHandle modelPin = GCHandle.Alloc(modelBytes, GCHandleType.Pinned);
GCHandle espeakPin = GCHandle.Alloc(espeakBytes, GCHandleType.Pinned);
GCHandle? configPin = null;
byte[]? configBytes = null;
if (configPath is not null)
{
configBytes = System.Text.Encoding.UTF8.GetBytes(configPath + "\0");
configPin = GCHandle.Alloc(configBytes, GCHandleType.Pinned);
}
try
{
return PiperNative.piper_create(
modelPin.AddrOfPinnedObject(),
configPin?.AddrOfPinnedObject() ?? IntPtr.Zero,
espeakPin.AddrOfPinnedObject());
}
finally
{
modelPin.Free();
espeakPin.Free();
configPin?.Free();
}
}
public ValueTask DisposeAsync()
{
if (_disposed)
return ValueTask.CompletedTask;
if (_synth != IntPtr.Zero)
{
PiperNative.piper_free(_synth);
_synth = IntPtr.Zero;
}
_disposed = true;
return ValueTask.CompletedTask;
}
}
@@ -0,0 +1,28 @@
using System.Runtime.InteropServices;
namespace Robovoice.Tts.LibPiper;
internal static class NativeDependencyLoader
{
private static int _loaded;
public static void EnsureLoaded()
{
if (Interlocked.CompareExchange(ref _loaded, 1, 0) != 0)
return;
string baseDir = AppContext.BaseDirectory;
string nativeDir = Path.Combine(baseDir, "runtimes", "win-x64", "native");
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && Directory.Exists(nativeDir))
{
if (!SetDllDirectory(nativeDir))
throw new InvalidOperationException(
$"SetDllDirectory failed for: {nativeDir}");
}
}
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetDllDirectory(string lpPathName);
}
+65
View File
@@ -0,0 +1,65 @@
using System.Runtime.InteropServices;
namespace Robovoice.Tts.LibPiper;
[StructLayout(LayoutKind.Sequential)]
internal struct PiperSynthesizeOptions
{
public int SpeakerId;
public float LengthScale;
public float NoiseScale;
public float NoiseWScale;
}
[StructLayout(LayoutKind.Sequential)]
internal struct PiperAudioChunk
{
public IntPtr Samples;
public nuint NumSamples;
public int SampleRate;
[MarshalAs(UnmanagedType.U1)]
public bool IsLast;
public IntPtr Phonemes;
public nuint NumPhonemes;
public IntPtr PhonemeIds;
public nuint NumPhonemeIds;
public IntPtr Alignments;
public nuint NumAlignments;
}
internal static class PiperNative
{
private const string LibName = "piper";
public const int PiperOk = 0;
public const int PiperDone = 1;
public const int PiperErrGeneric = -1;
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr piper_create(
IntPtr modelPath,
IntPtr configPath,
IntPtr espeakDataPath);
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern void piper_free(IntPtr synth);
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern PiperSynthesizeOptions piper_default_synthesize_options(IntPtr synth);
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern int piper_synthesize_start(
IntPtr synth,
IntPtr text,
in PiperSynthesizeOptions options);
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern int piper_synthesize_next(
IntPtr synth,
out PiperAudioChunk chunk);
[DllImport(LibName, CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr piper_version();
}
@@ -0,0 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<ProjectReference Include="..\Robovoice.Core\Robovoice.Core.csproj" />
</ItemGroup>
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<ItemGroup>
<None Include="runtimes\win-x64\native\piper.dll" CopyToOutputDirectory="PreserveNewest" Link="piper.dll" />
<None Include="runtimes\win-x64\native\onnxruntime.dll" CopyToOutputDirectory="PreserveNewest" Link="onnxruntime.dll" />
<None Include="runtimes\win-x64\native\onnxruntime_providers_shared.dll" CopyToOutputDirectory="PreserveNewest" Link="onnxruntime_providers_shared.dll" />
<None Include="runtimes\win-x64\native\espeak-ng-data\**\*" CopyToOutputDirectory="PreserveNewest" Link="espeak-ng-data\%(RecursiveDir)%(Filename)%(Extension)" />
</ItemGroup>
</Project>
+7
View File
@@ -0,0 +1,7 @@
<Solution>
<Project Path="Robovoice.App/Robovoice.App.csproj" />
<Project Path="Robovoice.Core/Robovoice.Core.csproj" />
<Project Path="Robovoice.Stt.File/Robovoice.Stt.File.csproj" />
<Project Path="Robovoice.Stt.Tcp/Robovoice.Stt.Tcp.csproj" />
<Project Path="Robovoice.Tts.LibPiper/Robovoice.Tts.LibPiper.csproj" />
</Solution>
+20
View File
@@ -0,0 +1,20 @@
[package]
name = "rvsttd"
version = "0.1.0"
edition = "2021"
[[bin]]
name = "rvsttd"
path = "src/main.rs"
[dependencies]
anyhow = "1"
cpal = "0.15"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
[build-dependencies]
bindgen = "0.71"
[profile.release]
opt-level = 3
+28
View File
@@ -0,0 +1,28 @@
use std::path::PathBuf;
fn main() {
let home = std::env::var("HOME").unwrap_or_else(|_| "/root".to_string());
let rvsttd_dir = PathBuf::from(&home).join(".rvsttd");
let lib_dir = rvsttd_dir.join("lib");
let include_dir = rvsttd_dir.join("include");
let header = include_dir.join("moonshine-c-api.h");
println!("cargo:rerun-if-changed={}", header.display());
println!("cargo:rustc-link-search=native={}", lib_dir.display());
println!("cargo:rustc-link-lib=dylib=moonshine");
println!("cargo:rustc-link-arg=-Wl,-rpath,{}", lib_dir.display());
let bindings = bindgen::Builder::default()
.header(header.to_str().unwrap())
.allowlist_function("moonshine_.*")
.allowlist_var("MOONSHINE_.*")
.allowlist_type("transcript.*|moonshine_option_t|speaker_span_t|transcript_word_t")
.derive_default(true)
.generate()
.expect("Unable to generate moonshine bindings");
let out_path = PathBuf::from(std::env::var("OUT_DIR").unwrap());
bindings
.write_to_file(out_path.join("moonshine_bindings.rs"))
.expect("Couldn't write bindings");
}
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
set -euo pipefail
RVSTTD_DIR="${HOME}/.rvsttd"
LIB_DIR="${RVSTTD_DIR}/lib"
INCLUDE_DIR="${RVSTTD_DIR}/include"
BUILD_DIR="${RVSTTD_DIR}/build"
ARCH="linux-x86_64"
# Fetch latest release tag from GitHub API
VERSION=$(curl -s https://api.github.com/repos/moonshine-ai/moonshine/releases/latest | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
if [ -z "${VERSION}" ]; then
echo "ERROR: could not fetch latest release tag from GitHub API"
exit 1
fi
PREBUILT_URL="https://github.com/moonshine-ai/moonshine/releases/download/${VERSION}/moonshine-voice-${ARCH}.tar.gz"
SOURCE_URL="https://github.com/moonshine-ai/moonshine/archive/refs/tags/${VERSION}.tar.gz"
echo "=== rvsttd setup ==="
echo "Latest Moonshine release: ${VERSION}"
echo "Target: ${RVSTTD_DIR}"
echo ""
mkdir -p "${LIB_DIR}" "${INCLUDE_DIR}" "${BUILD_DIR}"
# Step 1: Download prebuilt package (for libonnxruntime.so.1 + header)
PREBUILT_TGZ="${BUILD_DIR}/moonshine-voice-${ARCH}.tar.gz"
PREBUILT_EXTRACTED="${BUILD_DIR}/moonshine-voice-${ARCH}"
if [ ! -f "${LIB_DIR}/libonnxruntime.so.1" ]; then
echo ">>> Downloading prebuilt package (for libonnxruntime.so.1)..."
curl -L -o "${PREBUILT_TGZ}" "${PREBUILT_URL}"
mkdir -p "${PREBUILT_EXTRACTED}"
tar xzf "${PREBUILT_TGZ}" -C "${PREBUILT_EXTRACTED}" --strip-components=1
# Copy ONNX Runtime (prebuilt is fine — it has no glibc issue)
cp "${PREBUILT_EXTRACTED}/lib/libonnxruntime.so.1" "${LIB_DIR}/"
echo " Installed libonnxruntime.so.1"
# Copy the header (it's the same in source and prebuilt)
cp "${PREBUILT_EXTRACTED}/include/moonshine-c-api.h" "${INCLUDE_DIR}/"
echo " Installed moonshine-c-api.h"
else
echo ">>> libonnxruntime.so.1 already present, skipping prebuilt download"
fi
# Step 2: Download source
SOURCE_TGZ="${BUILD_DIR}/moonshine-source.tar.gz"
SOURCE_DIR="${BUILD_DIR}/moonshine-source"
if [ ! -d "${SOURCE_DIR}" ]; then
echo ">>> Downloading Moonshine source ${VERSION}..."
curl -L -o "${SOURCE_TGZ}" "${SOURCE_URL}"
mkdir -p "${SOURCE_DIR}"
tar xzf "${SOURCE_TGZ}" -C "${SOURCE_DIR}" --strip-components=1
fi
# Step 3: Build libmoonshine.so from source
CMAKE_BUILD="${BUILD_DIR}/cmake-build"
if [ ! -f "${LIB_DIR}/libmoonshine.so" ]; then
echo ">>> Building libmoonshine.so from source..."
# Point CMake at the prebuilt ONNX Runtime
ORT_LIB_DIR="${LIB_DIR}"
ORT_INCLUDE_DIR="${SOURCE_DIR}/core/third-party/onnxruntime/include"
# Patch CMake minimum version for older distros (Debian 11 ships 3.18)
find "${SOURCE_DIR}" -name CMakeLists.txt -exec sed -i 's/cmake_minimum_required(VERSION 3\.22\.1)/cmake_minimum_required(VERSION 3.18.4)/' {} +
# Remove -Werror (fails on third-party headers with older compilers)
sed -i 's/-Werror//' "${SOURCE_DIR}/core/CMakeLists.txt"
mkdir -p "${CMAKE_BUILD}"
cd "${CMAKE_BUILD}"
cmake "${SOURCE_DIR}/core" \
-DCMAKE_BUILD_TYPE=Release \
-DONNXRUNTIME_LIB_PATH="${ORT_LIB_DIR}/libonnxruntime.so.1" \
-DMOONSHINE_TTS_BUILD_ONNX=ON \
-DCMAKE_CXX_FLAGS="-I${SOURCE_DIR}/core/moonshine-tts/src"
make -j"$(nproc)" moonshine
cp "${CMAKE_BUILD}/libmoonshine.so" "${LIB_DIR}/"
cp "${SOURCE_DIR}/core/moonshine-c-api.h" "${INCLUDE_DIR}/"
echo " Installed libmoonshine.so (built from source)"
else
echo ">>> libmoonshine.so already present, skipping build"
fi
echo ""
echo "=== Setup complete ==="
echo "Library: ${LIB_DIR}/libmoonshine.so"
echo "Library: ${LIB_DIR}/libonnxruntime.so.1"
echo "Header: ${INCLUDE_DIR}/moonshine-c-api.h"
echo ""
echo "Next: cd rvsttd && cargo build --release"
echo "Then: ./target/release/rvsttd fetch # downloads the model"
echo "Then: ./target/release/rvsttd # starts the server"
+782
View File
@@ -0,0 +1,782 @@
use anyhow::{anyhow, Result};
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
use cpal::{SampleFormat, SampleRate};
use serde::Deserialize;
use std::collections::HashSet;
use std::ffi::CStr;
use std::io::{BufRead, BufReader, Write};
use std::net::{TcpListener, TcpStream};
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc;
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
include!(concat!(env!("OUT_DIR"), "/moonshine_bindings.rs"));
const SAMPLE_RATE: i32 = 16000;
const HEADER_VERSION: i32 = 30000;
const ARCH: u32 = 5; // MOONSHINE_MODEL_ARCH_MEDIUM_STREAMING
const BIND_ADDR: &str = "127.0.0.1:6996";
const MAX_TEXT_BYTES: usize = 1380;
// ─── helpers ──────────────────────────────────────────────────────────────
fn ts() -> String {
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default();
let secs = now.as_secs() % 86400;
let h = secs / 3600;
let m = (secs % 3600) / 60;
let s = secs % 60;
let ms = now.subsec_millis();
format!("{:02}:{:02}:{:02}.{:03}", h, m, s, ms)
}
fn log(msg: &str) {
eprintln!("[{}] {}", ts(), msg);
}
fn err_str(code: i32) -> String {
unsafe {
let s = moonshine_error_to_string(code);
if s.is_null() {
format!("error {}", code)
} else {
CStr::from_ptr(s).to_string_lossy().into_owned()
}
}
}
fn truncate_to_word(text: &str, max_bytes: usize) -> &str {
if text.len() <= max_bytes {
return text;
}
let cut = &text[..max_bytes.min(text.len())];
match cut.rfind(' ') {
Some(pos) => &text[..pos],
None => cut,
}
}
fn line_text(line: &transcript_line_t) -> String {
if line.text.is_null() {
return String::new();
}
unsafe { CStr::from_ptr(line.text) }
.to_string_lossy()
.into_owned()
}
fn unix_now() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
// ─── debug session recorder ──────────────────────────────────────────────
struct DebugRecorder {
dir: PathBuf,
audio: Vec<f32>,
log_lines: Vec<String>,
}
impl DebugRecorder {
fn new(base_dir: &PathBuf) -> Self {
let dir = base_dir.join(unix_now().to_string());
std::fs::create_dir_all(&dir).ok();
Self {
dir,
audio: Vec::new(),
log_lines: Vec::new(),
}
}
fn log_event(&mut self, event: &str) {
self.log_lines.push(format!("[{}] {}", ts(), event));
}
fn log_segment(&mut self, line: &transcript_line_t, prefix: &str) {
let text = line_text(line);
let event = format!(
"SEGMENT id={} prefix={} is_complete={} start={:.3}s duration={:.3}s text=\"{}\"",
line.id, prefix, line.is_complete, line.start_time, line.duration, text
);
self.log_event(&event);
}
fn add_audio(&mut self, samples: &[f32]) {
self.audio.extend_from_slice(samples);
}
fn save(self) {
// Write log
let log_path = self.dir.join("session.log");
match std::fs::File::create(&log_path) {
Ok(mut f) => {
for line in &self.log_lines {
writeln!(f, "{}", line).ok();
}
}
Err(e) => log(&format!("debug: failed to write log: {}", e)),
}
// Write WAV
let wav_path = self.dir.join("audio.wav");
match write_wav(&wav_path, &self.audio, SAMPLE_RATE as u32) {
Ok(()) => {
let secs = self.audio.len() as f64 / SAMPLE_RATE as f64;
log(&format!("debug: saved {} ({:.1}s, {} samples)", self.dir.display(), secs, self.audio.len()));
}
Err(e) => log(&format!("debug: failed to write wav: {}", e)),
}
}
}
fn write_wav(path: &PathBuf, samples: &[f32], sample_rate: u32) -> Result<()> {
let num_samples = samples.len() as u32;
let data_size = num_samples * 4; // f32 = 4 bytes
let file_size = 44 + data_size;
let mut f = std::fs::File::create(path)?;
// RIFF header
f.write_all(b"RIFF")?;
f.write_all(&(file_size - 8).to_le_bytes())?;
f.write_all(b"WAVE")?;
// fmt chunk
f.write_all(b"fmt ")?;
f.write_all(&16u32.to_le_bytes())?; // chunk size
f.write_all(&3u16.to_le_bytes())?; // IEEE float
f.write_all(&1u16.to_le_bytes())?; // mono
f.write_all(&sample_rate.to_le_bytes())?;
f.write_all(&(sample_rate * 4).to_le_bytes())?; // byte rate
f.write_all(&4u16.to_le_bytes())?; // block align
f.write_all(&32u16.to_le_bytes())?; // bits per sample
// data chunk
f.write_all(b"data")?;
f.write_all(&data_size.to_le_bytes())?;
// Convert f32 samples to little-endian bytes
let mut bytes = Vec::with_capacity(data_size as usize);
for &s in samples {
bytes.extend_from_slice(&s.to_le_bytes());
}
f.write_all(&bytes)?;
Ok(())
}
// ─── shared state ─────────────────────────────────────────────────────────
struct Shared {
writer: Mutex<TcpStream>,
session_id: u64,
transcriber_handle: i32,
debug_dir: Option<PathBuf>,
}
impl Shared {
fn send_msg(&self, prefix: &str, text: &str) {
let text = truncate_to_word(text, MAX_TEXT_BYTES);
let line = if text.is_empty() {
format!("{} {}\n", prefix, self.session_id)
} else {
format!("{} {} {}\n", prefix, self.session_id, text)
};
let mut writer = self.writer.lock().unwrap();
match writer.write_all(line.as_bytes()) {
Ok(_) => log(&format!("TX {} {} {}", prefix, self.session_id, text)),
Err(e) => log(&format!("TX failed: {}", e)),
}
}
}
// ─── session ──────────────────────────────────────────────────────────────
struct Session {
shared: Arc<Shared>,
stop_signal: Arc<AtomicBool>,
aborted: Arc<AtomicBool>,
transcriber: thread::JoinHandle<()>,
cpal_stream: Option<cpal::Stream>,
stream_handle: i32,
}
impl Session {
fn stop(mut self) {
// Signal transcriber to exit main loop, then wait for it to drain
// trailing audio + final flush. cpal stream stays alive during drain.
self.stop_signal.store(true, Ordering::SeqCst);
self.transcriber.join().ok();
// Now safe to kill ALSA — transcriber is done
self.cpal_stream.take();
unsafe { moonshine_free_stream(self.shared.transcriber_handle, self.stream_handle) };
}
fn abort(mut self) {
self.aborted.store(true, Ordering::SeqCst);
self.stop_signal.store(true, Ordering::SeqCst);
self.transcriber.join().ok();
self.cpal_stream.take();
unsafe { moonshine_free_stream(self.shared.transcriber_handle, self.stream_handle) };
}
}
fn start_session(shared: Arc<Shared>) -> Option<Session> {
let stream_handle = unsafe { moonshine_create_stream(shared.transcriber_handle, 0) };
if stream_handle < 0 {
log(&format!("create_stream failed: {}", err_str(stream_handle)));
return None;
}
let rc = unsafe { moonshine_start_stream(shared.transcriber_handle, stream_handle) };
if rc != 0 {
log(&format!("start_stream failed: {}", err_str(rc)));
unsafe { moonshine_free_stream(shared.transcriber_handle, stream_handle) };
return None;
}
let (tx, rx) = mpsc::channel::<Vec<f32>>();
let stop_signal = Arc::new(AtomicBool::new(false));
let aborted = Arc::new(AtomicBool::new(false));
let cpal_stream = match start_cpal(tx) {
Ok(s) => Some(s),
Err(e) => {
log(&format!("cpal failed: {}", e));
unsafe { moonshine_free_stream(shared.transcriber_handle, stream_handle) };
return None;
}
};
let shared_clone = shared.clone();
let stop_signal_clone = stop_signal.clone();
let aborted_clone = aborted.clone();
let transcriber = thread::spawn(move || {
transcriber_loop(shared_clone, rx, stop_signal_clone, aborted_clone, stream_handle);
});
Some(Session {
shared,
stop_signal,
aborted,
transcriber,
cpal_stream,
stream_handle,
})
}
fn transcriber_loop(
shared: Arc<Shared>,
rx: mpsc::Receiver<Vec<f32>>,
stop_signal: Arc<AtomicBool>,
aborted: Arc<AtomicBool>,
stream_handle: i32,
) {
let handle = shared.transcriber_handle;
let mut sent_ids: HashSet<u64> = HashSet::new();
// Debug recorder (if enabled)
let mut recorder = shared.debug_dir.as_ref().map(|_| DebugRecorder::new(
&shared.debug_dir.as_ref().unwrap().join(shared.session_id.to_string()),
));
if let Some(ref mut r) = recorder {
r.log_event(&format!("session {} started", shared.session_id));
}
// Main loop: process audio until stop_signal
while !stop_signal.load(Ordering::SeqCst) {
match rx.recv_timeout(Duration::from_millis(100)) {
Ok(chunk) => {
if let Some(ref mut r) = recorder {
r.add_audio(&chunk);
}
unsafe {
moonshine_transcribe_add_audio_to_stream(
handle, stream_handle,
chunk.as_ptr(), chunk.len() as u64,
SAMPLE_RATE, 0,
);
}
let mut t_ptr: *mut transcript_t = std::ptr::null_mut();
let rc = unsafe { moonshine_transcribe_stream(handle, stream_handle, 0, &mut t_ptr) };
if rc != 0 || t_ptr.is_null() {
continue;
}
if let Some(ref mut r) = recorder {
log_transcript_lines(r, t_ptr);
}
send_new_segments(&shared, t_ptr, &mut sent_ids, "P");
}
Err(mpsc::RecvTimeoutError::Timeout) => continue,
Err(mpsc::RecvTimeoutError::Disconnected) => break,
}
}
// Drain trailing audio from ALSA buffer (cpal stream still alive).
// Fixed 100ms window — cpal delivers every ~50ms (800 samples @ 16kHz),
// so this captures 1-2 more callbacks worth of trailing audio.
// Hold one chunk back so we can apply a fade-out to the very last one.
let drain_deadline = std::time::Instant::now() + Duration::from_millis(100);
let mut held_chunk: Option<Vec<f32>> = None;
while std::time::Instant::now() < drain_deadline {
match rx.recv_timeout(drain_deadline - std::time::Instant::now()) {
Ok(chunk) => {
// Feed previously held chunk to Moonshine + recorder (no fade)
if let Some(prev) = held_chunk.take() {
if let Some(ref mut r) = recorder {
r.add_audio(&prev);
}
unsafe {
moonshine_transcribe_add_audio_to_stream(
handle, stream_handle,
prev.as_ptr(), prev.len() as u64,
SAMPLE_RATE, 0,
);
}
}
held_chunk = Some(chunk);
}
Err(mpsc::RecvTimeoutError::Timeout) => break,
Err(mpsc::RecvTimeoutError::Disconnected) => break,
}
}
// Apply 5ms fade-out to the last chunk, then feed to both Moonshine and recorder
if let Some(mut chunk) = held_chunk.take() {
let fade_samples = (SAMPLE_RATE as usize * 25) / 1000; // 25ms
if chunk.len() > fade_samples {
let start = chunk.len() - fade_samples;
for i in 0..fade_samples {
let t = 1.0 - (i as f32 / fade_samples as f32);
chunk[start + i] *= t;
}
}
if let Some(ref mut r) = recorder {
r.add_audio(&chunk);
}
unsafe {
moonshine_transcribe_add_audio_to_stream(
handle, stream_handle,
chunk.as_ptr(), chunk.len() as u64,
SAMPLE_RATE, 0,
);
}
}
// If aborted (new session took over), skip final flush entirely
if aborted.load(Ordering::SeqCst) {
unsafe { moonshine_stop_stream(handle, stream_handle) };
log(&format!("Session {} aborted, skipping final flush", shared.session_id));
if let Some(mut r) = recorder {
r.log_event("aborted (new session took over)");
r.save();
}
return;
}
// Final flush
unsafe { moonshine_stop_stream(handle, stream_handle) };
let mut t_ptr: *mut transcript_t = std::ptr::null_mut();
let rc = unsafe { moonshine_transcribe_stream(handle, stream_handle, 0, &mut t_ptr) };
if rc == 0 && !t_ptr.is_null() {
let t = unsafe { &*t_ptr };
let mut new_segments: Vec<String> = Vec::new();
if let Some(ref mut r) = recorder {
log_transcript_lines(r, t_ptr);
}
for i in 0..t.line_count as usize {
let line = unsafe { &*t.lines.add(i) };
if line.text.is_null() || line.is_complete == 0 {
continue;
}
if !sent_ids.insert(line.id) {
continue;
}
let text = line_text(line);
if !text.is_empty() {
new_segments.push(text);
}
}
if new_segments.is_empty() {
shared.send_msg("F", "");
} else {
let last = new_segments.len() - 1;
for (i, text) in new_segments.iter().enumerate() {
let prefix = if i == last { "F" } else { "P" };
shared.send_msg(prefix, text);
if let Some(ref mut r) = recorder {
r.log_event(&format!("TX {} \"{}\"", prefix, text));
}
}
}
} else {
shared.send_msg("F", "");
if let Some(ref mut r) = recorder {
r.log_event("TX F (empty)");
}
}
if let Some(mut r) = recorder {
r.log_event("session ended");
r.save();
}
}
fn log_transcript_lines(recorder: &mut DebugRecorder, t_ptr: *const transcript_t) {
let t = unsafe { &*t_ptr };
for i in 0..t.line_count as usize {
let line = unsafe { &*t.lines.add(i) };
if line.text.is_null() {
continue;
}
recorder.log_segment(line, if line.is_complete != 0 { "complete" } else { "partial" });
}
}
fn send_new_segments(
shared: &Shared,
t_ptr: *const transcript_t,
sent_ids: &mut HashSet<u64>,
prefix: &str,
) {
let t = unsafe { &*t_ptr };
for i in 0..t.line_count as usize {
let line = unsafe { &*t.lines.add(i) };
if line.text.is_null() || line.is_complete == 0 {
continue;
}
if !sent_ids.insert(line.id) {
continue;
}
let text = line_text(line);
if text.is_empty() {
continue;
}
shared.send_msg(prefix, &text);
}
}
// ─── cpal ─────────────────────────────────────────────────────────────────
fn start_cpal(
tx: mpsc::Sender<Vec<f32>>,
) -> Result<cpal::Stream> {
let host = cpal::default_host();
let dev = host
.default_input_device()
.ok_or_else(|| anyhow!("no input device"))?;
let supported = dev
.supported_input_configs()?
.filter(|c| c.channels() <= 2 && c.min_sample_rate().0 <= 16000)
.min_by_key(|c| match c.sample_format() {
SampleFormat::F32 => 0,
SampleFormat::I16 => 1,
SampleFormat::U8 => 2,
_ => 99,
})
.ok_or_else(|| anyhow!("no suitable input config"))?;
let fmt = supported.sample_format();
let mut config = supported.with_max_sample_rate().config();
if config.channels > 1 {
config.channels = 1;
}
config.sample_rate = SampleRate(16000);
config.buffer_size = cpal::BufferSize::Fixed(800);
let err_fn = |e: cpal::StreamError| log(&format!("cpal error: {}", e));
let stream = match fmt {
SampleFormat::F32 => dev.build_input_stream(
&config,
move |data: &[f32], _: &_| {
let _ = tx.send(data.to_vec());
},
err_fn,
None,
)?,
SampleFormat::I16 => dev.build_input_stream(
&config,
move |data: &[i16], _: &_| {
let _ = tx.send(data.iter().map(|&x| x as f32 / 32768.0).collect());
},
err_fn,
None,
)?,
SampleFormat::U8 => dev.build_input_stream(
&config,
move |data: &[u8], _: &_| {
let _ = tx.send(data.iter().map(|&x| (x as f32 - 128.0) / 128.0).collect());
},
err_fn,
None,
)?,
_ => return Err(anyhow!("unsupported sample format {:?}", fmt)),
};
stream.play()?;
Ok(stream)
}
// ─── model fetch ──────────────────────────────────────────────────────────
#[derive(Deserialize)]
struct Manifest {
groups: Vec<ManifestGroup>,
}
#[derive(Deserialize)]
struct ManifestGroup {
#[allow(dead_code)]
base_url: String,
files: Vec<ManifestFile>,
}
#[derive(Deserialize)]
struct ManifestFile {
name: String,
url: String,
size: Option<u64>,
}
fn fetch_model(model_dir: &str) -> Result<()> {
let dest = PathBuf::from(model_dir);
log(&format!("Fetching medium-streaming-en model to {}", dest.display()));
let lang = std::ffi::CString::new("en").unwrap();
let opt_name = std::ffi::CString::new("model_arch").unwrap();
let opt_value = std::ffi::CString::new("5").unwrap();
let mut options = [moonshine_option_t {
name: opt_name.as_ptr(),
value: opt_value.as_ptr(),
}];
let mut json_ptr: *mut i8 = std::ptr::null_mut();
let rc = unsafe {
moonshine_get_stt_dependencies(
lang.as_ptr(),
options.as_mut_ptr(),
options.len() as u64,
&mut json_ptr,
)
};
if rc != 0 || json_ptr.is_null() {
return Err(anyhow!("moonshine_get_stt_dependencies failed: {}", err_str(rc)));
}
let json_str = unsafe { CStr::from_ptr(json_ptr) }
.to_string_lossy()
.into_owned();
unsafe { moonshine_free_buffer(json_ptr as *mut std::ffi::c_void) };
let manifest: Manifest = serde_json::from_str(&json_str)?;
std::fs::create_dir_all(&dest)?;
let mut total_files = 0;
let mut total_bytes: u64 = 0;
for group in &manifest.groups {
for file in &group.files {
let dest_path = dest.join(&file.name);
if dest_path.exists() {
log(&format!(" SKIP {} (already exists)", file.name));
continue;
}
if let Some(parent) = dest_path.parent() {
std::fs::create_dir_all(parent)?;
}
log(&format!(" GET {}", file.url));
if let Some(expected) = file.size {
log(&format!(" {} bytes", expected));
total_bytes += expected;
}
let status = std::process::Command::new("curl")
.arg("-sSL")
.arg("-o")
.arg(&dest_path)
.arg(&file.url)
.status()?;
if !status.success() {
return Err(anyhow!("curl failed for {}", file.url));
}
total_files += 1;
}
}
log(&format!("Done: {} files, ~{} MB", total_files, total_bytes / (1024 * 1024)));
log(&format!("Model directory: {}", dest.display()));
Ok(())
}
// ─── main ─────────────────────────────────────────────────────────────────
fn main() -> Result<()> {
let mut args = std::env::args().skip(1);
let home = std::env::var("HOME").unwrap_or_else(|_| "/root".to_string());
let default_model = format!("{}/.rvsttd/model", home);
let first = args.next();
if first.as_deref() == Some("fetch") {
let model_dir = args.next().unwrap_or_else(|| default_model.clone());
return fetch_model(&model_dir);
}
let mut args = first.into_iter().chain(args);
let mut model_dir = default_model;
let mut debug = false;
while let Some(a) = args.next() {
match a.as_str() {
"--model-dir" | "-m" => {
model_dir = args.next().unwrap_or(model_dir);
}
"--debug" => {
debug = true;
}
"--help" | "-h" => {
println!("Usage: rvsttd [--model-dir DIR] [--debug]");
println!(" rvsttd fetch [DIR]");
println!("Listens on TCP {}", BIND_ADDR);
println!("Model: medium-streaming (Moonshine)");
println!("--debug: save session audio + transcript log to ~/.rvsttd/debug/");
println!("'rvsttd fetch' downloads the English medium-streaming model");
return Ok(());
}
_ => return Err(anyhow!("unknown arg: {}", a)),
}
}
let debug_dir = if debug {
let d = PathBuf::from(&home).join(".rvsttd").join("debug");
std::fs::create_dir_all(&d).ok();
log(&format!("Debug mode enabled — sessions saved to {}", d.display()));
Some(d)
} else {
None
};
let model_path = std::fs::canonicalize(&model_dir)
.unwrap_or_else(|_| std::path::PathBuf::from(&model_dir));
log(&format!("Loading model from {}...", model_path.display()));
let c_dir = std::ffi::CString::new(model_path.to_str().unwrap()).unwrap();
let transcriber_handle = unsafe {
moonshine_load_transcriber_from_files(
c_dir.as_ptr(),
ARCH,
std::ptr::null(),
0,
HEADER_VERSION,
)
};
if transcriber_handle < 0 {
return Err(anyhow!("failed to load model: {}", err_str(transcriber_handle)));
}
log(&format!("Model loaded (handle {})", transcriber_handle));
let listener = TcpListener::bind(BIND_ADDR)?;
log(&format!("STT server listening on TCP {}", BIND_ADDR));
let mut current_session: Option<Session> = None;
for stream in listener.incoming() {
let stream = match stream {
Ok(s) => s,
Err(e) => {
log(&format!("accept failed: {}", e));
continue;
}
};
stream.set_nodelay(true).ok();
log(&format!("Client connected: {}", stream.peer_addr().map(|a| a.to_string()).unwrap_or_else(|_| "?".to_string())));
let writer_stream = stream.try_clone()?;
let reader = BufReader::new(stream);
for line in reader.lines() {
let line = match line {
Ok(l) => l,
Err(_) => break,
};
let line = line.trim();
log(&format!("RX {}", line));
// ON <session>
if let Some(rest) = line.strip_prefix("ON ") {
let new_session_id: u64 = rest.parse().unwrap_or(0);
if let Some(s) = current_session.take() {
log(&format!("Aborting session {} for new session {}", s.shared.session_id, new_session_id));
s.abort();
}
let shared = Arc::new(Shared {
writer: Mutex::new(writer_stream.try_clone()?),
session_id: new_session_id,
transcriber_handle,
debug_dir: debug_dir.clone(),
});
log(&format!("PTT on session {}", new_session_id));
match start_session(shared) {
Some(s) => current_session = Some(s),
None => log("Failed to start session"),
}
}
// OFF <session>
else if let Some(rest) = line.strip_prefix("OFF ") {
let off_session: u64 = rest.parse().unwrap_or(0);
if let Some(s) = current_session.as_ref() {
if s.shared.session_id == off_session {
log(&format!("OFF session {}", off_session));
if let Some(s) = current_session.take() {
s.stop();
}
} else {
log(&format!("OFF session {} (stale, current={}), ignoring", off_session, s.shared.session_id));
}
} else {
log(&format!("OFF session {} (no active session), ignoring", off_session));
}
}
else {
log(&format!("Unknown command: {}", line));
}
}
log("Client disconnected");
if let Some(s) = current_session.take() {
s.abort();
}
}
Ok(())
}