§ 01
PROBLEM
Children's speech is different — and most ASR pretends otherwise.
Speech-therapy researchers need phoneme-level transcripts, but pediatric corpora are small, noisy, and expensive to label. Out-of-the-box ASR trained on adult speech fails hard: disfluencies, vocal-tract length differences, and atypical articulation push Phoneme Error Rate (PER) to 80% on CSLU Kids — essentially unusable for downstream clinical work.
The manual alternative — expert phonetic annotators — takes ~45 minutes per minute of audio. At the scale a single clinic generates, that's not a pipeline; that's a bottleneck.
The goal wasn't higher accuracy on adult benchmarks. It was making the long tail of real speech cheap to annotate.
§ 02
APPROACH
A two-model pipeline with confidence-weighted fallback.
AutoPhon combines a fine-tuned Wav2Vec 2.0 phoneme head (fast, coarse) with a Whisper forced-alignment pass (slow, accurate) gated by a per-segment confidence score. High-confidence segments exit after Wav2Vec; low-confidence ones get the expensive second pass.
STEP 01
Wav2Vec 2.0
Phoneme CTC head · fine-tuned
STEP 02
Confidence gate
Softmax margin · per-segment
STEP 03
Whisper align
Low-confidence only
STEP 04
Merge + clean
Human-readable TextGrid
Fine-tuning was staged: first on MyST (largest, clean) for 20 epochs, then joint on CSLU Kids + PhonBank with a per-corpus loss scaling to prevent the smallest corpus from dominating. LR warmup + cosine decay, AdamW, gradient checkpointing on a single A100.
§ 03
RESULTS
Below 15% PER on held-out data.
Phoneme Error Rate · CSLU Kids held-out
Wav2Vec · pretrained
66.1%
More importantly, 90% of segments exit after the Wav2Vec pass — the Whisper fallback is reserved for the hard 10%, keeping end-to-end cost low enough to run on a single A100 overnight for a week of clinic audio.
§ 04
NEXT
Porting to the edge.
Next thread: running the Wav2Vec stage on Jetson Orin Nano for real-time on-device phoneme feedback in speech therapy apps. Targeting sub-100ms latency with outlier-aware INT8 quantization — see the parallel quantization benchmarks project.
