when i first knew i wanted to align audio to text, i reached for aeneas, which was built for this exact problem/product (https://www.readbeyond.it/aeneas/)
however, it drifted by hundreds of seconds deep into chapters and emits no confidence signal, so i couldn't eval it cleanly
so i switched to a neural acoustic model trained with ctc (meta's massively multilingual speech aligner, it ships inside torchaudio): it outputs a probability for each character at every frame of audio, and viterbi decoding finds the most probable path that spells exactly our text, with a per-word probability
that probability is the gate: a book ships read-along only if its median confidence and coverage clear a bar. moby dick's didn't, so it's honestly text-only
when i first knew i wanted to align audio to text, i reached for aeneas, which was built for this exact problem/product (https://www.readbeyond.it/aeneas/)
however, it drifted by hundreds of seconds deep into chapters and emits no confidence signal, so i couldn't eval it cleanly
so i switched to a neural acoustic model trained with ctc (meta's massively multilingual speech aligner, it ships inside torchaudio): it outputs a probability for each character at every frame of audio, and viterbi decoding finds the most probable path that spells exactly our text, with a per-word probability
that probability is the gate: a book ships read-along only if its median confidence and coverage clear a bar. moby dick's didn't, so it's honestly text-only