The difference using an mp3 seems to be smaller: yap seems to use about the same time but fluidaudio seems to take twice as long. Do you happen to know why?
Investigated this and it turned out to be an amusing bug: audio decoding was happening three times instead of just once lol. I've put up a PR to remove the wasteful redundant decoding:
With the updated PR code, ran a test comparing transcribing (using Parakeet V3) a 1 hr stereo 44.1 kHz mp3 vs the same audio in 16 kHz mono wav format. The result was about 21.3% slower with the mp3 vs the wav, i.e. that's the overhead of decoding + resampling.
Currently the decoding + resampling is done up front. If it was done in a pipelined fashion with the inference, that overhead can be eliminated. This is what I did in a recent app I made:
It uses FluidAudio as well, but I forked it and replaced the audio decoding code to (a) use mpg123 instead of the native Apple API and (b) do audio decoding and inference in a pipelined fashion. These two changes effectively eliminated the overhead. mpg123 is quite a bit faster than the native Apple API at mp3 decoding (has some very optimized arm64 assembly routines), and the pipelining ensures that the inference is never starved by the mp3 decoding.
Contributing this pipelined setup to FluidAudio would be good.
I use fluidaudiocli and it's unfortunate that it doesn't support streaming (e.g. from a named pipe); that would have been an easy workaround to both the pipelining problem and the faster-decoder problem.
That app was exactly what I was looking for, something like SponsorBlock but for podcasts but I suppose using AI for finding the ads works too. Any chance it'll release on Android?
- parakeet usually runs on Bfloat16. NPU doesn't support that
- CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay
- Parakeet latency is fine but "fine" may not be good enough for Apple's UX team.
- CPU increases power consumption over dedicated float blocks
So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!