Research|Audiobook pipeline — reference & findings

docs

Audiobook pipeline — reference & findings

Last modified 11 October 2026 · view source

Agent-facing reference for the local TTS audiobook pipeline. The working code, text, and a shorter README live in projects/audiobook/. This doc is the durable record of what was learned and why each setting is what it is, so a future agent doesn’t re-derive it.

Last updated: 2026-06-28.

Goal

Publication-quality local audiobooks for tolstoy.life works — one .m4b per work, beside the existing EPUBs. Long-term: an incremental nightly job that generates audio only for new/changed works and writes _generated/audio/<work>.m4b.

Status

Stack

Findings (the hard-won ones)

  1. Use synthesize, not infinite. This was the fix that took the result from “improved” to “amazingly good.” The infinite command (built for long-form streaming) adds a processing step that mangles the ends of long sentences — a pre-final pause plus a high-pitched last word. The plain synthesize command renders a single sentence cleanly. An A/B on two long sentences confirmed it; synthesize clips were also ~0.5 s tighter.

  2. Synthesize one sentence at a time; splice silence between. Kokoro has no SSML. The W3C standard for pauses is SSML <break>, but Kokoro can’t read markup — so explicit per-unit synthesis + inserted silence is the correct workaround, not a hack.

  3. Pause lengths (seconds): sentence 0.45, paragraph 0.85, Part break 2.0. Grounded in: ACX/Audible section breaks 2–2.5 s; TTS engine defaults ~0.3/0.4 s (too short for long-form); prosody research (period ≈ 0.8–1.2 s total incl. the voice’s own fall). Never insert silence at commas — chopping intra-sentence destroys Kokoro’s pitch contour and sounds robotic.

  4. Spell out Part headers. Kokoro reads Part II as “Part two-eye.” The build converts Part <Roman>. → “Part One/Two/…”; the M4B chapter list keeps Roman numerals (they read better visually).

  5. Spell out money. $1.40 → “one dollar forty” (done in the flow text, not the source — the source keeps the real characters).

  6. Respell the verb “live” → “liv”. Kokoro otherwise says /laɪv/ (as in “live broadcast”). Whole-word only; “lives/lived/living” untouched.

  7. Auto-join page-break-split paragraphs. OCR/extraction splits some paragraphs mid-sentence. Rule: a paragraph not ending in . ! ? " ) … was split by a page break → glue it to the next. Replaces hardcoded per-chapter join indices.

  8. M4B must mux with -movflags +faststart or iOS Books silently refuses to import it (moov atom must precede mdat).

  9. Mastering chain (the LibriVox-fatigue fix — what makes spoken word pleasant): highpass=f=70,deesser=i=0.4,loudnorm=I=-19:TP=-2:LRA=7. Output 44.1 kHz, 128k AAC. Kokoro is 24 kHz native — that’s the fidelity ceiling; higher bitrate is wasted.

  10. Per-call model reload (~5 s). kokoro-tts-tool reloads Kokoro on every invocation, so the ~410-sentence book takes ~40 min cold. Fine for now (the build is resumable). For the nightly pipeline, batch synthesis in one process.

  11. Merge very short sentences into a neighbour — but at the SEGMENT level, not the build. A 2–4 word clip synthesized alone rises instead of falling — Kokoro can’t land a declarative cadence without runway (“But we are wrong.”). The original merge_short lived in build_audiobook.py and was removed: merging after segments.json changed the sentence count, which broke the read-along contract (1 sentence = 1 SMIL <par> = 1 highlight). The correct home is reader/segment.py (merge_speech_groups), which glues the pair before segments.json is written — so clips, timing, and SMIL all stay consistent and the pair simply highlights as one read-along unit (2026-07-04). Which sentences merge is an explicit id list (reader/speech.py MERGE_FORWARD), not a word-count rule — a blanket “short sentence” rule would wrongly flatten legitimate rising questions (“Why is this?”, “Whence this dreadful perversity?”). It does not fold “Part One.” into the first sentence (that would span an <h2>+<p>, which one highlight can’t) — headings that rise when spoken alone are an accepted limit; finding #4’s Roman→word conversion still applies.

  12. Pronunciation respellings live in a SUBS dict in the build, applied at synth time so the source text stays faithful (same place as live→liv). Validated by ear (auditioned against the raw reading): Labouchere→ Labooshair, Radischeff→Rahdeeshef (Radishchev), Yasnaya Poliana→ Yasnaya Polyahna, and the scripture ref (Matt, xxiii. 27, 28)→spoken “Matthew twenty-three, verses twenty-seven and twenty-eight”. Novikoff, Komaroff, Decembrists, Toynbee, Parnell were auditioned and left as-is.

  13. Split over-long sentences so Kokoro doesn’t speed-read them (a ~90-word sentence gets compressed and rushed). The build now reads flow text generated with flow_preprocess.py --split-long 45 — sentences over 45 words split once at the comma nearest the middle. Same fidelity trade-off as the semicolons (adds a full stop Tolstoy didn’t write), accepted for listenability.

  14. Probe pronunciations with espeak-ng IPA before auditioning by ear. espeak-ng -v en-gb -q --ipa "<word>" prints exactly what the g2p will hand the voice — respelling candidates can be screened in seconds without synthesizing anything. Found this way (2026-07-03, voice-notes pass): a word-initial kv- cluster is unreachable — English phonotactics has none, so espeak letter-names the K (“Kvas”/”kvass”/any kv-spelling → two-syllable “kay-v…”). Dictionary English kvass is /kvɑːs/ (like “kvetch”), but the closest the engine can be spelled into is quahss → /kwɑːs/ — accepted. espeak’s raw-phoneme notation ([[kvA:s]]) works in espeak itself but is mangled by the phonemizer layer inside kokoro-tts-tool, and the CLI exposes no phoneme input — a true /kv/ would need driving kokoro-onnx directly with hand phonemes (one-off, cache-fragile; not done).

  15. Measure prosody with parselmouth — don’t guess by ear alone. pip install praat-parselmouth; Sound(wav).to_pitch() gives the F0 contour, short-time RMS gives pause lengths. The objective probe for “does this rise or fall” (companion to finding #16’s espeak IPA probe). Settled two things this way (2026-07-04): (a) a rising terminal appositive — “…support us, their parasites.” rose to ~149 Hz on the closing tag; swapping the comma for an em-dash in speech only (“support us — their parasites.”, page keeps the comma) drops it to ~111 Hz so it falls. (b) Comma pause length is a fixed Kokoro constant (~140 ms) — no lever. Measured: slowing to --speed 0.93 keeps commas at ~145 ms (it stretches the words); commas→semicolons gives ~147 ms (Kokoro treats them the same). The only thing that lengthens a comma is a full stop, which resets pitch — so it’s a per-spot tool (finding #5’s lists, the “God,” pause), never a global default.

  16. A dash gets the same short pause as a comma (~0.14 s). Fine inside a sentence, too little between the items of a headline-style list such as The Kingdom of God’s chapter summaries. Those paragraphs are listed in reader/speech.py DASH_STOPS and their dashes become full stops in speech only (measured ~0.25–0.4 s after the change, 2026-10-11) — the same full-stop lever as finding 5’s lists.

  17. The text we narrate is the 1905 Mayo/Tchertkoff translation, verbatim. A line-by-line check against the Russian original confirmed the phrasings that sound odd (“clambered out” = слезла; “exhaustion of the strength of nations” = истощение сил народов; “Socialistic organisation” = социалистическое устройство — Tolstoy’s own word for the educated classes’ engineered future society, not a translator’s narrowing) are faithful, not transcription errors. The only real defects were transcriber artifacts, now fixed in chapters/: the “Transcription/Markup: Andy Carloff” credit line (removed), a stray editorial “(?)” in ch03 (removed), and a missing space in ch04.

  18. Sideloaded audiobooks don’t iCloud-sync (by design). Manual import only: AirDrop → Files → Share → Copy to Books, or Finder cable sync. EPUBs/PDFs sync; audiobooks never have.

Model landscape (June 2026), for the record

Better-than-Kokoro free models exist (Chatterbox, OpenAudio S1, Qwen3-TTS, Orpheus) but they win on expressiveness, which is a liability for neutral long-form Tolstoy (they “act,” wander, and hallucinate words more). Kokoro ties commercial engines on clean-read naturalness (UTMOS ~4.48) and is notably clean — the right tool here. SSML only really exists in cloud engines; the one free high-quality SSML path is edge-tts, rejected because it’s online + an undocumented endpoint (this pipeline is deliberately local).

How to run

From projects/audiobook/:

Open questions / next

Decisions log