<!DOCTYPE html><htmllang="en"data-astro-cid-5hce7sga><head><metacharset="utf-8"><metaname="viewport"content="width=device-width, initial-scale=1"><title>Local Voice AI: What Actually Works for TTS and Voice Cloning — dd0c.net</title><linkrel="icon"href="/favicon.ico"><style>[data-astro-cid-5hce7sga],[data-astro-cid-5hce7sga]:before,[data-astro-cid-5hce7sga]:after{box-sizing:border-box}body{font-family:system-ui,-apple-system,sans-serif;margin:0;background:#f8f9fa;color:#333;line-height:1.6}nav[data-astro-cid-5hce7sga]{background:#fff;border-bottom:2pxsolid#3294D2;padding:01.5rem;display:flex;align-items:center;gap:0}nav[data-astro-cid-5hce7sga].brand[data-astro-cid-5hce7sga]{display:flex;align-items:center;gap:.5rem;text-decoration:none;margin-right:1.5rem;padding:.75rem0}nav[data-astro-cid-5hce7sga].brand[data-astro-cid-5hce7sga]img[data-astro-cid-5hce7sga]{height:32px;width:auto}nav[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga]{color:#333;text-decoration:none;padding:.75rem.85rem;font-size:.95rem;transition:color.15s}nav[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga]:hover{color:#3294d2}nav[data-astro-cid-5hce7sga].spacer[data-astro-cid-5hce7sga]{flex:1}nav[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga].external{color:#3294d2}main[data-astro-cid-5hce7sga]{max-width:760px;margin:2remauto;padding:01.25rem}footer[data-astro-cid-5hce7sga]{margin-top:3rem;padding:1.25rem;text-align:center;font-size:.875rem;color:#666;border-top:1pxsolid#e0e0e0;background:#fff}footer[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga]{color:#3294d2;text-decoration:none}footer[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga]:hover{text-decoration:underline}h1[data-astro-cid-5hce7sga]{font-size:1.75rem;color:#1a1a1a}h2[data-astro-cid-5hce7sga]{font-size:1.2rem;margin-top:1.75rem;color:#1a1a1a}a[data-astro-cid-5hce7sga]{color:#3294d2}p[data-astro-cid-5hce7sga]{margin:.6rem0}@media(max-width:600px){nav[data-astro-cid-5hce7sga]{flex-wrap:wrap;padding:0.75rem}nav[data-astro-cid-5hce7sga].brand[data-astro-cid-5hce7sga]{margin-right:.5rem}nav[data-astro-cid-5hce7sga]a[data-astro-cid-5hce7sga]{padding:.6rem.5rem;font-size:.875rem}}
</style></head><bodydata-astro-cid-5hce7sga><navdata-astro-cid-5hce7sga><aclass="brand"href="/"data-astro-cid-5hce7sga><imgsrc="/logo-white.svg"alt="dd0c.net"width="40"height="40"data-astro-cid-5hce7sga></a><ahref="/"data-astro-cid-5hce7sga>Home</a><ahref="/about"data-astro-cid-5hce7sga>About</a><ahref="/services"data-astro-cid-5hce7sga>Services</a><divclass="spacer"data-astro-cid-5hce7sga></div><aclass="external"href="https://github.com/ddoc"target="_blank"rel="noopener"data-astro-cid-5hce7sga>GitHub</a></nav><maindata-astro-cid-5hce7sga><aclass="back"href="/"data-astro-cid-gjtny2mx>← All posts</a><h1data-astro-cid-gjtny2mx>Local Voice AI: What Actually Works for TTS and Voice Cloning</h1><pclass="post-meta"data-astro-cid-gjtny2mx>2026-03-24</p><divclass="post-body"data-astro-cid-gjtny2mx><p>I spent the last few weeks setting up local text-to-speech and voice cloning for a personal AI assistant. Here’s what actually happened — not what the docs promise.</p>
<h2id="piper-tts-the-reliable-workhorse">Piper TTS: The Reliable Workhorse</h2>
<p><ahref="https://github.com/rhasspy/piper">Piper</a> runs in Docker on my TrueNAS server via the <ahref="https://github.com/rhasspy/wyoming">Wyoming protocol</a>. Setup was straightforward. Pick a voice model, point it at a port, done.</p>
<p>The speed genuinely surprised me. On a machine that’s primarily a NAS, not a compute server, Piper generates speech at realtime speed or better. I wrote a small proxy that wraps the Wyoming protocol in an OpenAI-compatible <code>/v1/audio/speech</code> endpoint so anything that speaks that API can use it.</p>
<p>The catch: Piper has one voice at a time. I tried using it for a two-host podcast and both hosts sounded identical. For single-voice use cases — notifications, voice replies, alerts through a Sonos speaker — it’s excellent. For anything requiring distinct voices, you need something else.</p>
<h2id="qwen3-tts-voice-cloning-impressive-but-painful">Qwen3-TTS Voice Cloning: Impressive But Painful</h2>
<p><ahref="https://huggingface.co/Qwen/Qwen3-TTS">Qwen3-TTS</a> runs on my M1 Pro MacBook via <ahref="https://github.com/Blaizzy/mlx-audio">mlx-audio</a>. The pitch: give it a voice sample and it generates new speech in that voice.</p>
<p><strong>It’s slow.</strong> About 0.3x realtime on M1 Pro on average, though it varies wildly — I benchmarked five runs and got anywhere from 0.09x to 0.41x. A 12-second clip takes 30–50 seconds on a good run, but can spike to over two minutes when thermally throttled. I set it up as a LaunchAgent so the server stays running, but the speed makes it impractical for anything interactive. Short voice messages are fine. A 20-minute podcast is not happening.</p>
<p><strong>The 24kHz gotcha cost me hours.</strong> Input reference audio must be 24000 Hz mono WAV. If you feed it 16kHz audio — which is what most voice recorders produce — the output is truncated garbage. No error message. No warning. Just broken audio. I resampled everything with ffmpeg and the problem disappeared, but there was nothing in the docs pointing to this.</p>
<p><strong>Style steering makes it worse.</strong> The model has an <code>--instruct</code> mode for controlling tone and delivery. Every time I used it for voice cloning, the output quality dropped. Removing it and letting the model match the reference audio naturally produced better results every time.</p>
<p><strong>Sample quality matters more than sample length.</strong> The docs suggest a few seconds of audio is enough. Technically true, but a clean 10-second conversational clip produces noticeably better clones than a noisy 3-second snippet. I got the best results from natural speech — someone talking normally, not reading a script.</p>
<p>I cloned a few voices for fun. Some worked well on the first try. Others took multiple attempts with different reference clips before they sounded right. It’s not a “give it any audio and get a perfect clone” situation. It’s more like “give it good audio and you’ll get a recognizable approximation.”</p>
<p><strong>Voice replies.</strong> Someone sends a voice message on Telegram, <ahref="https://github.com/openai/whisper">Whisper</a> transcribes it, the AI generates a text response, and Piper converts it to speech. The reply goes back as a voice note. Piper handles this well because speed matters more than voice variety.</p>
<p><strong>Podcast generation.</strong> I built a pipeline that researches news, writes a two-host script, and generates audio. I tried Piper first but both hosts sounded the same. Qwen was too inconsistent for 20 minutes of audio — generation speed varies 4x between runs. I ended up using a cloud TTS API with two distinct voices for this. Local TTS isn’t there yet for multi-voice long-form content.</p>
<p><strong>Home automation alerts.</strong> Piper speaks weather alerts and system notifications through a Sonos speaker. Fast enough that the latency is imperceptible.</p>
<p><strong>Voice cloning for short clips.</strong> Qwen handles personalized greetings, short jokes in cloned voices, that kind of thing. Anything under 15 seconds where the two-minute generation time is acceptable.</p>
<p>I want to get Piper running with multiple voice models on different ports. That would solve the podcast problem without needing cloud TTS. Piper supports dozens of voices — I just need to run more than one instance.</p>
<p>For Qwen, the speed is the bottleneck. If Apple ships faster Neural Engine silicon or the model gets optimized further, voice cloning could become practical for longer content. Right now it’s a novelty for short clips.</p>
<p>The setup cost me nothing beyond hardware I already owned. If you have a machine that runs Docker, <ahref="https://github.com/rhasspy/piper">Piper</a> is a 15-minute setup. Qwen requires more patience — both for the initial configuration and for waiting on every generation.</p></div></main><footerdata-astro-cid-5hce7sga><ahref="/privacy"data-astro-cid-5hce7sga>Privacy Policy</a> ·
