1 Motivation

I like sending friends links to things I’ve written. What I noticed is that almost nobody reads them.

Not out of disinterest — when I asked directly, the honest answer was closer to I don’t have a block of time to sit down and read this right now. And that’s a real constraint, not an excuse. Reading a long-form post demands a contiguous slot of attention: you can’t do it while driving, cooking, or walking the dog. Even for me, reading my own drafts requires setting other things aside.

A podcast doesn’t have that problem. It fills time that’s otherwise unusable — a commute, a chore, a walk. If the barrier to “reading” my blog was really a barrier of format, not content, then the fix wasn’t to write shorter posts. It was to give the same content a second, ambient form.

That’s the entire premise behind this project: take the blog’s existing bilingual content, generate natural-sounding narration in my own cloned voice, and let readers listen instead of read, without changing what gets written or how.

2 Project Overview

Goal: generate natural-sounding, voice-cloned narration of blog posts — both Chinese and English — entirely offline, on a home-lab GPU (a Tesla P40, 24GB, Pascal architecture), then publish the audio to Cloudflare R2 so the live site can stream it alongside the text.

The constraint that shaped every decision below was offline, local inference. Not because commercial TTS APIs are bad — I genuinely don’t know what a subscription plus the customization I’d need would cost, though I doubt it’s cheap — but because the marginal cost of running this on hardware I already own is close to zero. The only real expense is electricity. That trade-off — a slower, more failure-prone development path in exchange for a near-zero long-run operating cost — is a decision I’d make again, and it’s the reason most of this devlog is about debugging a model rather than calling an API.

The scope by the numbers: 42 blog posts, published in both languages, meaning 84 audio files to generate for the full back catalog. Each post runs five to ten minutes of narration depending on length. Local generation on the P40 was estimated at roughly 30 hours of GPU time to clear the entire backlog.

3 Starting Point: F5-TTS and the First Wall

The original plan used F5-TTS for both languages, cloning a voice zero-shot from a short reference clip. This is the simplest possible approach: no training, just a few seconds of reference audio and a text prompt.

Infrastructure went up first — the inference VM, a Cloudflare Tunnel to reach it, R2 credentials for storage — and English narration came out acceptable on the first attempt. Chinese did not. Voice cloning quality on F5-TTS wasn’t good enough for the Chinese voice, and no amount of reference-clip adjustment closed the gap. That result pushed the project toward a fine-tuning-based approach instead of zero-shot cloning, on the reasoning that a model trained specifically on my voice should generalize better than one prompted with a few seconds of it.

That reasoning turned out to be only half right, for reasons that took the next several days to understand.

4 Pivot to GPT-SoVITS

GPT-SoVITS fine-tunes a small model on a few minutes of a single speaker’s voice, split into two stages:

SoVITS — the acoustic/timbre model, responsible for what it sounds like.
GPT (also called T2S) — an autoregressive model that turns text into semantic tokens, responsible for what it says.

The workflow was: dataset formatting (slicing the reference recording, ASR-based transcription, BERT/HuBERT feature extraction) → fine-tuning SoVITS, then GPT → inference.

Chinese worked first. Trained on the v2Pro architecture using a Chinese reference recording sliced into short clips, and after iterating on inference parameters — slice method, temperature, pause duration — the result was good enough to sign off on: “中文版本可以了.”

English didn’t. The first attempt used only about a minute of reference recording and produced incomplete sentences and dropouts. A fresh, cleaner three-minute recording fixed most of that — but exposed two new problems that took much longer to solve than the original one.

5 Two Bugs, Two Very Different Kinds of Fixes

5.1 The shallow one: missing pauses

GPT-SoVITS didn’t reliably pause at colons, semicolons, or em-dashes. The fix was mechanical: normalize these to commas before synthesis, since commas were a punctuation mark the model consistently paused on, in both language paths:

_CN_PAUSE_NORMALIZE = {"：": "，", "；": "，", "——": "，"}
_EN_PAUSE_NORMALIZE = {";": ",", ":": ","}

This fully resolved the pause issue for both languages, in one pass, and never came back. It’s worth naming as a category of its own precisely because it’s the exception — most of what follows was not this simple.

5.2 The deep one: dropped words

Independent of punctuation, generation would occasionally drop an entire word or clause. Non-deterministically. Not linearly correlated with text length. The kind of bug that resists a clean hypothesis because it doesn’t reproduce on demand.

Building a round-trip verification harness

The first move was to stop relying on ad-hoc listening and build a check: generate audio, transcribe it back with funasr SenseVoice, diff the transcription against the source text with difflib.SequenceMatcher, and treat any leftover (“deleted”) words as a dropped-word failure. On failure, retry with a different text-splitting strategy — five variants, from no-split to splitting on every punctuation mark — and keep the best of up to five attempts, requiring exactly zero missing words before accepting a result.

This felt like the right instinct: replace subjective listening with an objective, automatable check.

One production run still dropped the word “adult” entirely, despite passing verification. That specific failure traced to a percentage-based tolerance that was too loose, and tightening it to exact zero missing words closed that hole.

But a second, deeper blind spot surfaced afterward and was never fully resolved: the ASR model’s own language model can “heal” a truncated word into a complete one. Concretely — “responsibility,” “authority,” and “legitimacy” each lost their final syllable, specifically when the word landed right before a comma-induced pause inside a generation chunk. And the round-trip check reported all three as fully present, every time, because SenseVoice heard the intended word rather than the actual clipped audio it was given. Its language model filled the gap the same way a human reader’s brain autocorrects a typo — except here, that correction was actively hiding the bug from the one system built to catch it.

From that point forward, human listening was the only reliable check for this specific failure class. The automated harness stayed useful for everything else, but I stopped trusting it as a sole gate for this bug.

Dead ends, each backed by an actual test

Several hypotheses got tested and rejected — not assumed away, but disproven by generating audio and listening to it:

Attempt	Result
Hard-split on every comma (`cut5`)	Fixed one word’s completeness but caused a different word to disappear entirely, and garbled others (“legitimacy” → “legitims”)
Slow generation down 10% (`speed_factor=0.9`)	Made truncation worse, including whole words vanishing
Assume 3 minutes of training audio is insufficient; record more	Rejected on documentation, not assumption — GPT-SoVITS’s own docs state roughly one minute is enough
Switch model version v2Pro → v3/v4, since the docs claim “fewer missing words”	Retrained both languages via LoRA on the same existing recordings. The round-trip check reported clean results, but my own ears still caught the same words slightly clipped — and Chinese quality was a regression against v2Pro

The documentation check on training-data length is worth pausing on. “Maybe I just don’t have enough training audio” was the intuitively obvious next move — it’s the fix that feels like more effort equals more quality. Checking the project’s own docs before acting on that intuition redirected the whole investigation, because it ruled out the one hypothesis that would have cost the most time to test and would still have been wrong.

The architectural root cause

Version-switching didn’t fix it because it couldn’t have. GPT-SoVITS’s word content is produced by the autoregressive GPT stage — stochastic sampling, token by token, the same mechanism that causes hallucination and skipping in text LLMs, applied here to speech tokens. That stage’s core sampling behavior is essentially unchanged across v1 through v4. The documented v3/v4 improvements (“重复漏字更少”) are concentrated in the SoVITS/vocoder decoder stage — audio fidelity — not the token-generation stage that decides which words get spoken. That’s why switching versions never reliably fixed anything, and why v4 even sounded worse on this dataset: v3/v4 also lean more heavily on the timbre of a single reference clip, trading away some of the natural variation v2Pro captured from the fuller training set.

Once the bug was understood at this level, it stopped being a bug I could tune my way out of.

6 When Tuning Stops Being the Right Question

This is the actual turning point of the project, and it’s a different kind of decision than anything above it.

Every fix up to this point had operated inside GPT-SoVITS: different splitting strategy, different speed, different training data, different model version. Once the root cause was understood as architectural — the failure lived in an unconstrained autoregressive sampling stage, not in data quantity or inference parameters — no amount of tuning inside that architecture could resolve it. The right question changed from how do I tune this model to which architectures don’t have this failure mode by construction.

Comparing published benchmarks: CosyVoice2 (WER ~2.7% on LibriSpeech, still autoregressive — same failure class, just less frequent), GPT-SoVITS (WER ~5.1%), and IndexTTS2 (Bilibili, 2025) — state-of-the-art WER among the three, and notably the first autoregressive TTS with explicit, millisecond-level duration control, purpose-built to reduce exactly this failure class. IndexTTS2 also defaults to beam search (num_beams=3) with a high repetition penalty (10.0), instead of GPT-SoVITS’s pure sampling — directly cutting the variance that lets an unlucky sample clip a word.

The bigger structural win: IndexTTS2 requires no fine-tuning at all. It clones a voice zero-shot from a few seconds of reference audio, using a speaker-embedding extractor pretrained on a large multi-speaker corpus — the same principle as face recognition identifying a new face from one photo, without retraining the underlying model. That eliminates the entire slice/transcribe/train pipeline GPT-SoVITS required, and with it, most of the debugging surface described above.

One thing I checked before committing: bilibili’s custom license for the model only requires a separate paid license above 100M MAU or RMB 1B in annual revenue. A personal blog is nowhere near that threshold.

7 Getting IndexTTS2 Running on a Pascal GPU

None of these problems were about the model itself — all environment friction, the unglamorous kind that eats an afternoon without teaching you anything transferable:

Disk full mid-download. The VM’s root disk hit 100%; resolved by expanding it.
deepspeed extra failed to build. No CUDA toolkit (nvcc) installed on the box, only the driver. Skipped — it’s optional.
torch 2.8, the project’s pinned default, completely dropped Pascal (sm_61) kernel support — CUDA error: no kernel image is available for execution on the device. torch.cuda.get_arch_list() confirmed the installed wheel only supported sm_70 and above. Fixed by downgrading to torch==2.4.0+cu118, matching the version GPT-SoVITS was already running successfully on the same card.
uv run silently re-syncs the environment against the project’s lockfile before every invocation, which kept undoing the manual torch downgrade. Fixed by invoking .venv/bin/python3 directly instead of routing through uv run.

8 Validating Quality, and Tuning by Elimination

The first zero-shot test reused a few seconds sliced from the same recordings already made for GPT-SoVITS — no new recording, no training. It was dramatically better than months of GPT-SoVITS tuning, on the first try.

One process note worth keeping: the first Chinese test text wasn’t actually verbatim blog content — it was a continuation improvised to match the reference clip’s style — and it was audibly noticeable: it “felt generated.” Re-running against verbatim paragraphs from an actual post confirmed the result held on real content, which mattered, because a model can sound convincing on text it was implicitly primed for and less convincing on genuinely unseen prose.

With a working baseline established — fixed reference clip, all generation parameters at default, no emotion conditioning — every further change was tested against that fixed baseline, one variable at a time, rather than combinatorially:

Auto emotion-from-text (use_emo_text=True) — rejected. Text-inferred emotion skewed heavily toward anger, and the whole voice came across as unhinged.
Manual 8-dimensional emotion vectors (three hand-designed variants aiming for a measured, analytical tone matching the blog’s writing style) — rejected. Described as sounding like two different voices forced together. The baseline, with no emotion conditioning at all, stayed the best result.
Reference clip duration (5.6s / 15s / 30s, same start point, Chinese) — no improvement over the original 5.6-second clip.
Reference clip position (three points across a ~130-second Chinese recording) — none beat the original.
Reference clip position, English (six points across a ~196-second recording) — the clip taken at the 130-second mark was clearly best.
Chinese speaking rate — IndexTTS2 has no native speed parameter, so post-hoc time-stretching via ffmpeg’s atempo was tried (0.95x / 0.92x, pitch-preserving). The unmodified original speed still won.
English pronunciation clarity — the delivery felt too crisply enunciated, missing the elision and liaison of native connected speech. This one didn’t resolve into a parameter fix at all: it’s an inherent property of the reference clip’s own speaking style, recorded deliberately and clearly for training purposes. IndexTTS2 clones delivery style along with timbre, not just spectral characteristics — so a clearly-enunciated reference produces a clearly-enunciated clone, by design, not by a tunable knob. This was accepted as a reasonable trade-off, since clearer enunciation also helps non-native listeners on a bilingual blog.

The rejected emotion-conditioning attempts are the more interesting result of this section. The failure mode in both cases wasn’t “not as good as baseline” — it was “sounds like two voices.” That’s a qualitatively different signal from a parameter simply being sub-optimal; it indicates the direction was wrong, not that the magnitude needed adjusting. Recognizing that distinction is what closed off further tuning in that direction rather than trying five more emotion-vector variants.

The confirmed baseline for both languages — reference clip plus generation parameters — was locked and recorded for reproducibility before moving to full-article generation.

9 Full-Article Generation and the Last Two Fixes

Generating complete narration for the pilot post, in both languages, with real content read verbatim, surfaced two smaller issues:

A citation containing an em-dash character hit an unknown BPE token. Logged as a known cosmetic edge case at the time, not fixed immediately.
English output was roughly 16dB quieter than Chinese. Traced to the English reference clip itself having been recorded about 16dB quieter than the Chinese one — the model faithfully reproduces the reference clip’s recording loudness, not just its timbre. Fixed with an EBU R128 loudness-normalization pass (ffmpeg loudnorm, target -16 LUFS) applied to both outputs going forward, independent of whatever level any future reference clip happens to be recorded at.

10 Building the Frontend Player

With two finished audio files sitting in R2, the next question was how a static blog turns into something with playable, seekable narration — and, just as importantly, how it does not turn into something that starts talking on its own.

The one hard product requirement going in: clicking a paragraph must never start playback by itself. A reader browsing in silence should never be surprised by sudden audio because a click happened to land on a sentence. This single constraint shaped the entire interaction model — a paragraph or table-of-contents click only seeks within narration the reader has already started via an explicit play button. Before that button is pressed, clicks do nothing.

Built as six pieces, deliberately scoped to the pilot post first:

A Cloudflare Pages Function (GET /api/audio?slug=&lang=) reading the R2 binding directly and returning availability, audio URL, timestamps, and duration.
A bottom-anchored player bar (play/pause, progress, elapsed/duration, speed select, close), CSS-hidden by default and only unhidden once the API confirms narration exists for that post. It exposes a small global registry so other scripts — paragraph highlighting, the table of contents — can find the active audio element and its timestamps without tight coupling.
A script that assigns an id to every heading and paragraph so each can be highlighted and clicked, listens for playback time updates to know which segment is currently playing, and gates every click-to-seek behind whether playback has actually started — the mechanism enforcing the no-autoplay-on-click requirement.
A table-of-contents extension that dispatches a custom event on click, letting a TOC click jump the audio to that section without the TOC component needing to know anything about how audio playback works internally.
Layout wiring so every post page — not just the pilot — automatically gets a player bar the moment its audio exists in R2, with no further frontend deploys needed per post going forward.

11 Four Bugs Caught Before Shipping

CORS. The first design returned a second, separate timestamps URL pointing at the custom R2 domain, to be fetched independently on the client. That fetch failed outright — the custom domain has no CORS headers configured, and browsers enforce CORS on fetch()/XHR even though they don’t enforce it on <audio src> elements. Fixed by having the server-side function read the timestamps file directly via the R2 binding and embed it in the single JSON response, turning two requests (one of them cross-origin) into one same-origin request.

A heading/paragraph id mismatch. This one was the most informative of the four. The frontend’s first id-assignment logic assumed headings and paragraphs were numbered by two independent counters. Inspecting the real timestamps data showed headings are actually numbered by the combined block count at insertion time, while paragraphs use a separate, paragraph-only counter — a detail of the backend’s block-numbering scheme that wasn’t visible from the frontend side at all. Fixed with a single running counter shared across both, plus a second counter for paragraphs alongside it.

Playback surviving a language switch. Switching from the English to the Chinese version of a post (or back) left the previous audio playing, with no visible control able to stop it. Root cause: the Audio object lives on the window, independent of the DOM elements the page-transition system swaps out on navigation. Swapping in a new player bar doesn’t touch the old, still-playing audio instance. Fixed by pausing and clearing the player registry on the page-transition lifecycle event that fires immediately before the DOM is replaced.

A speed selector overflowing the mobile player bar. On a narrow viewport, the speed dropdown pushed past the right edge of the rounded bar. The dropdown’s own width wasn’t actually the cause — the real problem was the progress slider’s default browser minimum-width preventing the flex row from shrinking at all, pushing everything after it off the edge together. Fixed by allowing that element to shrink below its default minimum.

12 Two Incidents During the Full-Catalog Backfill

With the pilot post’s integration verified, the next step was generating narration for the rest of the back catalog — 39 more posts, both languages — and pushing it live.

Before starting an unattended overnight run, a structural scan of all files caught three markdown patterns the text parser didn’t yet handle correctly: a bare blockquote marker being read aloud literally, a markdown table read as a string of pipes and dashes, and a divider glued to the next line’s heading, with no blank line between, merging into one garbled block. All three were fixed and reverified — zero suspect blocks across every file — before the batch launched. Two further problems only surfaced once it was actually running.

12.1 Editing a file on disk doesn’t change a running process’s behavior

Reviewing the batch the next morning, several already-generated posts with blockquotes still had the raw markdown symbol spoken aloud — despite the fix having been deployed and verified with a fresh test script before the batch started. The fix was real and correct. The problem was that the generation service runs as a long-lived process that had already imported the old parsing module into memory at startup, and the language runtime caches imported modules — overwriting the file on disk has no effect on a process that already loaded it. The standalone test script that “confirmed” the fix ran as a fresh process each time, so it correctly picked up the new file — which is exactly what masked the fact that the actual running service hadn’t. Fixed by restarting the service, which forces a fresh import, then regenerating the handful of posts produced with the stale parser in the interim.

This is a generic trap, not specific to this project: verifying a fix in a short-lived process tells you the fix is correct, not that it’s deployed. Those are two separate claims, and it’s easy to conflate them when the test that confirms the first one looks identical to what would confirm the second.

12.2 A paragraph-click seek bug traced to DOM traversal depth

Spot-checking newly published posts turned up a consistent bug: clicking paragraph N seeked to paragraph N-1’s audio instead — but only on posts containing lists or blockquotes, never on the pilot post or other list-free posts. That inconsistency was the clue. Root cause: the frontend’s id-assignment walk only visits an article’s direct children. A markdown list renders as a wrapper element containing list items, and a blockquote similarly wraps its content — neither the wrapper nor its children are a heading or a plain paragraph, so a shallow scan skips the entire structure without incrementing any counter. But the backend parser counts each list item and each blockquote as its own paragraph block — they’re just blank-line-separated markdown blocks like any other paragraph, with no special treatment. Every list or quote in a post therefore left the frontend’s paragraph counter permanently one block behind the backend’s, compounding with each further list or quote encountered later in the same post. Fixed by explicitly assigning ids to individual list items and to the blockquote element as a single unit, matching the backend’s counting exactly.

This bug and the em-dash BPE issue from earlier are the same underlying category, at different layers: a text-processing assumption that was correct for the pilot post’s specific structure quietly stopped being correct once the input distribution widened.

13 A Near-Miss with GPU Memory

Before starting the overnight batch, a routine health check showed nearly the entire 24GB of the P40 in use — despite only the IndexTTS2 service being an intentional, active workload. The remainder belonged to two GPT-SoVITS processes, its web UI and API server, left running idle since the engine pivot and never shut down. With essentially no headroom left, an unattended multi-hour batch run would have been one memory spike away from an out-of-memory crash partway through the night. After confirming both leftover processes were genuinely unused rather than something else depending on them, stopping them freed enough of the GPU to leave comfortable headroom for the run ahead.

The mistake wasn’t technical — it was procedural. Nothing about the pivot to IndexTTS2 required leaving the old service running; it simply wasn’t shut down when it stopped being needed, and stayed invisible until a batch job’s memory pressure exposed it.

14 Where It Stands Now

The timeline end to end ran three days. Day one was requirements and planning — writing down what I actually wanted this feature to do before touching any code, specifically to avoid losing the thread partway through a multi-day, multi-pivot build. Day two was the technical work described above: the model pivots, the debugging, and an overnight run generating narration for the full back catalog. Day three was review and the bug-fixing described in the sections above.

The backfill target is 84 audio files — 42 posts, two languages each — with individual narrations running five to ten minutes depending on post length, and a projected local-compute cost of roughly 30 GPU-hours to clear the entire catalog. The only real operating cost is electricity.

Two categories of known issue remain open as of writing:

Player bar responsiveness. The desktop layout is solid; mobile still needs further polish beyond the single overflow fix described in §11.
Markdown residue in narration. The three patterns fixed before the backfill (blockquote markers, tables, glued dividers) don’t cover every markdown construct across 42 posts written over more than a year. Further symbols will surface as edge cases the parser hasn’t seen yet, in the same way the em-dash BPE token and the list/quote counting bug did.

Neither blocks the feature from being live; both are being worked through as they surface.

I recorded A/B comparison audio throughout — emotion-vector variants, reference-clip position tests, before/after the pause-normalization fix — but I’m not publishing those clips here. They were working material for making decisions, not polished samples.

15 Reflections

Architecture beats data quantity, for this class of bug. More training data was the obvious-seeming fix and it was wrong. The actual fix was picking a model whose word-generation stage isn’t unconstrained autoregressive sampling. No amount of additional data would have touched a failure mode located one layer below where the data feeds in.

Verify claims against primary documentation, not assumption. The belief that three minutes of training audio wasn’t enough was disproven by GPT-SoVITS’s own documentation stating one minute suffices — and that single check redirected the entire investigation toward the actual cause, instead of burning a day re-recording reference audio that was never the problem.

Automated verification has blind spots that must be disclosed, not papered over. The ASR round-trip check couldn’t catch a truncation that its own language model could “fix” through inference. That’s not a flaw unique to this harness — it’s a structural risk in using any model-based verifier to check another model’s output, since both can share the same failure mode. Human review stayed part of the pipeline for exactly this reason, and I don’t think it can be fully removed.

Editing code and deploying code are different claims, and it’s easy to conflate them. The systemd module-caching incident cost a night’s worth of regeneration not because the fix was wrong, but because the test that “confirmed” it ran in a context — a fresh process — that couldn’t have caught the actual gap between disk and memory.

Change one variable at a time. Across a large tunable surface — reference clip choice, duration, position, emotion vector, sampling parameters — testing each change against a fixed, previously-confirmed baseline kept every result attributable to a single cause. It’s slower than testing combinations. It’s also the only way any of the negative results above were actually trustworthy.

If I had to name the single decision that mattered most, it’s the one in §6: recognizing that tuning had stopped being the right category of action, and that the question had to change from how to fix GPT-SoVITS to whether GPT-SoVITS’s architecture could ever be fixed at all. Everything productive after that point followed from asking the right question, not from working harder on the wrong one.

Blog Audio Reading System — A Voice-Cloned Narration Pipeline