docs/research/lib — shared research tooling

Canonical tooling for corpus-dive and the docs/research/ thematic sweeps.

• extract_tei.py — resolves tolstoydigital-TEI editorial markup into grep-able Russian prose. Usage: python3 extract_tei.py <path-to-xml> [substring] [--notes=auto|off|force] [--choice=legacy|reg|orig|both]. Requires lxml. This is the canonical copy; dives reference it rather than forking it.
• Note-encoded body recovery. Most documents carry their text in <p> body elements, with <note>/<noteGrp> holding footnotes that are stripped. A handful of PSS documents instead encode the document’s own text inside <noteGrp type="comments"> (the body has no <p> at all) — e.g. the 1910 Explanatory Note to Tolstoy’s will (v82_305, PSS Tom 82 pp. 227–231), which otherwise extracted to the opener alone. --notes=auto (the default) recovers that text only when the body has no real <p>/<closer> prose, so a normal extraction can never be polluted with its footnotes; corpus-wide, auto changes the output of exactly 3 files (v82_305, v78_174, v78_177). Genuine nested footnotes (e.g. <note resp="volume_editor" xml:id="…">) are still stripped. Recovered paragraphs are tagged [note] and announced by a # note-body: banner — they mix the document’s own text with editorial commentary, so verify against the source PDF. --notes=off forces the legacy strip-everything behaviour; --notes=force additionally dumps the comments apparatus after a normal body (for reading the editorial commentary).
• Pre-reform orthography (--choice). Pre-1918 texts encode old/new spelling as <choice><orig>стараго</orig><reg>старого</reg></choice>. The default legacy mode drops these pairs (the historical gap that gutted Tom 58 / 1880-era diaries and letters) — but now prints a # warning: to stderr naming the count so a run is never silently gutted. --choice=reg resolves to the modern-orthography <reg> reading and is recommended for every Prophet-period (pre-1918) extraction; --choice=orig keeps the pre-reform reading; --choice=both emits reg [orig] for collation. Editorial <sic>/<corr> pairs are always resolved to <corr>, independent of this flag. This supersedes the per-dive _reg_extract.py helper (e.g. lords-prayer/extracts/_reg_extract.py), which is left in place only as that dive’s provenance record.
• verify_quotes.py — mechanical byte-fidelity gate for a dive’s dossier. Loads every evidence[].quoteRu, opens the named extract file, and asserts the (whitespace-normalised) quote appears verbatim in it — turning Phase 5’s core credibility check into one deterministic command instead of an LLM judgement call. Also checks declared facsimile: files exist and warns (soft) when a quoteEn lacks the (working English) label. Honours author elision marks: leading/trailing ellipsis (mid-sentence excerpt, bare or bracketed) is stripped before matching; an internal […]/[...] splits the quote into fragments that must each appear, in order — but a bare internal … stays literal (it can be Tolstoy’s own). A mismatch report names the exact diverging word. Requires PyYAML. Usage: python3 docs/research/lib/verify_quotes.py docs/research/<slug>/dossier.yaml [--quiet]. Exit 0 = PASS, 1 = FAIL (mismatch or missing facsimile), 2 = usage/parse error.
• build_evidence_index.py — cross-dive aggregator. Walks every docs/research/*/dossier.yaml and emits an entity-keyed evidence index so wiki ingestion reuses verified research instead of re-collating it across dives (the same entity recurs in many dives — Chertkov in all five). For each entity (keyed by the slug of wikilinkTarget, == the eventual wiki/works slug) it resolves that dive’s evidenceRefs to full evidence rows qualified by dive, collates and de-dupes visuals (by url, so the same image cached under different per-dive paths merges with an alsoInDives list), unions sources, and re-derives vaultStatus live against website/src/wiki/ and website/src/works/ (stub = prose body < 60 words, or a draft < 120). Writes docs/research/evidence-index/evidence-index.yaml (machine) + index.md (human; renders to index.html via serve.py --build-only). A lint block flags unresolved evidenceRefs, name/wikiType conflicts (e.g. Birukoff/Biryukov, Maude person/translator), works-routed-not-wiki, vaultStatus drift, and zero-evidence research gaps. Output is deterministic (byte-identical on rebuild). Requires PyYAML. Usage: python3 docs/research/lib/build_evidence_index.py [--check] [--quiet] [--research-dir PATH]. Exit 0 = built (or --check clean), 1 = --check found broken links (unresolved refs), 2 = usage/parse error.
• test-verify-quotes.sh — regression check for the above: asserts a clean quote passes, a tampered body fails, a boundary-ellipsis excerpt passes, a valid internal [...] elision passes, and a corruption hidden behind an elision still fails. Run: bash test-verify-quotes.sh (prints PASS).
• test-extract-tei.sh — regression check for the above: asserts v82_305 recovers the operative will provision and the recovery banner, that its nested volume-editor footnote stays stripped, and that a normal letter (v78_170) keeps its body while its editorial apparatus is not leaked. Run: bash test-extract-tei.sh (prints PASS).
• test-corpus-dive-queue.sh — --dry-run command-generation check for the queue runner.
• corpus-dive-queue.sh — overnight queue runner for corpus-dive. Spawns a fresh claude -p session per theme; skips blank lines and # comments; writes a batch summary to docs/research/_batch-YYYY-MM-DD.md. Pass --skip-permissions to append --allow-dangerously-skip-permissions to each invocation (opt-in, OFF by default; required for truly unattended runs; flag name current as of Claude Code 2.1.x — the older --dangerously-skip-permissions is no longer accepted). Use --dry-run to print commands without executing. Usage: corpus-dive-queue.sh --themes <file> [--model <tier>] [--skip-permissions] [--dry-run]. Headless permission behaviour (re-probed 2026-05-29): the skill is found, triggered, and arg-parsing works (EXIT=0). In claude -p, a tool that needs permission and isn’t allowlisted is cleanly denied — the run continues/exits, it does not hang — when the session runs with a controlled posture (--setting-sources project + an --allowedTools/settings allow-list); read-only “safe” commands are auto-approved. This script sets no such posture (it inherits your user settings), so for truly unattended batches still pass --skip-permissions or supply a .claude/settings.json allow-list — otherwise a run may deny the writes the dive needs, or behave per whatever your user permission posture auto-approves.

Single source of truth (forks removed)

This is the only copy of extract_tei.py. Four dives (copyright-renunciation/, christian-communism-socialism/, doukhobors/, tolstoyanism-christian-anarchism/) used to carry byte-identical forks that had to be re-synced by hand on every change; those forks were deleted and each dive’s index.md now links to this copy via a relative path (../lib/extract_tei.py). New dives should reference this file rather than copying it. (Resolves the §17 cleanup in docs/superpowers/specs/2026-05-29-corpus-dive-design.md.)