I installed this on Mac OS but the program always fails with:
Uncaught Exception:
Error: Could not get code signature for running application
at m(/Applications/Youka.app/Contents/Resources/app/.webpack/main /index.js:1:12481)
at App.<anonymous> (/Applications/Youka.app/Contents/Resources/app/.webpack/main/index.js:1:14365)
at App.emit (events.js:215:7)
> Align text to voice (the hardest part) using some private api
That's also the part that would be most interesting to have explained. Is it language-agnostic? After all, the title says "in any language", but I can't think of any text-audio alignment algorithms that don't require a language-specific model. (Unless you just count characters and assume they map linearly to time, which I'd expect to go very badly.)
Having worked for many years in a linguistics research lab where we spent a lot of money paying people to edit and align subtitles and audio transcripts, and having largely written what was at the time the most sophisticated subtitle-and-transcript editing tool available, I can confirm: counting characters and mapping them linearly to timespan, even after isolating vocals, does indeed go very poorly. And much worse when there's singing involved.
Alternately, since you say speech recognition isn't "even close", I might try going the other way--doing text-to-speech on the audio stream, attempting to align the two speech tracks, and the back-porting the timecodes from audio alignment onto the text.
But that seems a lot more complicated... so, unlikely.
A way to cheat that would probably work good enough most of the time would be to spectrographic analysis on the audio stream to identify syllables, and then similarly just count syllables in the known text and line those up. That works better the more consistent your spelling system is, though, and still requires language-specific modelling. If you actually want to do a decent job cross-linguistically, you'd need in the general case a dictionary for every supported language listing syllable counts for each word (because not everybody's orthography is transparent enough to make simple models like counting character sequences work).
If you actually have a fully language-agnostic algorithm for aligning text to audio that's actually decently accurate, though, that's gotta be worth at least a Master's degree in computational linguistics, 'cause on the face of it it doesn't seem to me (who has such a Masters degree) that it should even theoretically be possible.
The way I'd do it is to use an existing speech recognition system with a large number of language models available (like CMU Sphinx--but probably not CMU Sphinx, 'cause I don't think there are decent openly-available models for 108 different language for Sphinx; maybe MicroSoft's Azure speech to text API or IBM's Watson speech recognition or something like that) to produce a rough transcript with timecodes, and then meet in the middle--use the timecodes from speech recognition, and the known-good text from whatever lyrics you already found, and reduce it to a text-to-text alignment problem so you can match up the ASR timecodes to the known-good text. First pass, I'd probably try an LCS match on the two text streams, but if that wasn't good enough, I'm sure there are better algorithms in the bioinformatics literature.
Examining the source, it looks like alignment is done via an HTML form data submission to 'https://api.audioai.online/split-align'. Manually visiting that website, however, is not very informative... the entire text of http://audioai.online is
Audio AI API
Split voice from audio
Sync voice to text
contact
Hey there! First of all, I want to tell you that the app is fantastic. I used the earlier version of this, when it was a website, from your previous HN post. And once again the alignment works quite well in my experience, as does the isolation.
In the future, it would be great to have a "portable" version of this for Windows that doesn't install anything. It's annoying to open up an app, and have it install itself without any warning or user consent. You could just release a .zip file with the build as an option.
You can distribute a .zip file which includes the statically linked build of FFmepeg: https://ffmpeg.zeranoe.com/builds/ . Then just call it locally. There's no need to install it system-wide.
Personally I love karaoke, but looking at the repo and the website gave me no information whatsoever about this project. Maybe that's something you can work on? In the meantime I found this article, which reads quite positive: https://www.theverge.com/tldr/2020/2/19/21144452/youtube-you...
Is there a way to manually provide the lyrics? I have a substantial collection of songs in Chinese and Taiwanese, and it would be really helpful to use this to help me make lyrics videos for Pingtype. When I tried, I got this error:
Ooops, some error occurred :(
Error: name 'espeakng_supported_langs' is not defined
I'll look into aeneas to see if that can give the API-level technical tools that I need - thank you for explaining that part in the other comments!
Note that it won't work for Taiwanese (I assume Hokkien) unless you add the necessary support to espeak-ng.
If your lyrics are in Peh-oe-ji, you'll need to define how the romanization maps to phonemes. You may be able to get some inspiration for that from the definitions for Mandarin and Cantonese. Though I just looked at the "phonology" section on Wikipedia https://en.wikipedia.org/wiki/Taiwanese_Hokkien#Phonology and the tone sandhi rules look a lot more complex than any other Sinitic language I know.
Tones are difficult, so I encode those as colours. Adding code to espeak-ng sounds very difficult. Most of the songs are in Mandarin though, so I'll try those first.
oh, I had the same idea and started working here https://github.com/redraw/karaoke-machine days after Deezer's spleeter was released, but stopped while searching for a way to sync the lyrics. thx! I'll try it out
edit: Swapped youtube URL to Peertube for Content ID claims issues.