[rmunn]

Looks very cool. One thing I have been looking for, which this doesn't seem to cover (at least I didn't see any mention of IPA in the model documentation), is a way to transcribe unknown languages phonetically, using the International Phonetic Alphabet to spell them (sound-based spelling rather than meaning-based spelling). I know several linguists doing research on minority languages (fewer than 10,000 speakers in some cases), which are small enough that they will never have enough effort made towards training language-specific models in that language.

Are there models I'm not aware of that are trained for this task? Taking audio in an unknown language, and rather than identifying the language, just transcribing the sounds to IPA? That would not be useful to most people, but it would be a Godsend to many, many linguists working with minority languages around the world.

[shenberg]

We take a lot of shortcuts when speaking, it's actually much harder to transcribe phonemes than to transcribe words, even when aware of the language being spoken. Some models have been trained for the task (e.g. look at https://huggingface.co/spaces/KoelLabs/IPA-Transcription-EN ), but the error rate is really high.

[rmunn]

There are, broadly, two kinds of audio recordings that linguists want to transcribe. One is native speakers telling traditional stories, where they're speaking naturally and taking the natural shortcuts (such as "wanna" and "gonna" in English). The other is native speakers reading words (or short example sentences) very carefully and distinctly, so that the linguist can listen to the recording over and over to learn how to pronounce the word right. In those recording, they'll say "want to" and "going to" rather than "wanna" and "gonna".

Thanks for the pointer; I'll check out that model and see if it handles the "slowly and carefully" type of recording better than the "natural speaking" type. (And depending on what kinds of errors the model makes, even the recordings where it makes errors can prove useful: for example, a linguist studying regional variations in speech would want the model to produce the IPA for "gonna" rather than "going to").

[uoaei]

Dialects degenerate phonemes that would otherwise occupy identity relations between different utterances of the same "word" (word/concept mutable hyperobject as is the standard in any socially-relevant spoken language) which would be a bit of irony in this thought experiment since common knowledge dictates that more data samples must be present in the dataset (not less, as in rarely-spoken languages) to associate separate pronunciations of utterances representing the same underlying concept. However very-rarely-spoken languages probably don't have distinct dialects since so much focus is put on mutual intelligibility with the few members of the group that remain fluent in that language. It's not outside the realm of possibility that small speaking communities nonetheless fractionate into dialectical specialities but that seems increasingly unlikely as the fervor for preserving/recognizing dying languages increases, and global instant communication continues to become more commonplace.

Example: Schwabisch is wild and would be phonetically transcribed very differently from Hochdeutsch which is its ostensible language progenitor (technically more a cousin than an ancestor in the lineage of language evolution), but if the goal is merely to focus the model purely on phonetic transcription then you can add additional post-processing layers which map sounds to core concepts shared across dialects for actual translation. But I like your idea of interacting with the intermediate elements to familiarize yourself at least with the phonetic patterns, we humans are still thinkers enough to infer patterns of grammar and semantics from these building blocks just as we have done for the entire history of the species/lineage before written representations of language came along (relatively late -- evidence of script cropped up only once civilization had centralized to a sufficient degree to make economics non-local and non-trivial).

tl;dr the big words: it's not til you collect enough spoken samples of the dead(ish/dying) language being spoken that the local idiosyncracies are discovered, luckily linguists are smart enough to probably anticipate and certainly post-process language snippets to grasp the common structures for this or that given language.

[rmunn]

> ... very-rarely-spoken languages probably don't have distinct dialects ...

That's true if you mean "very rarely spoken" literally, as in even the native speakers don't get to use it very often. But many languages aren't widely spoken (such as only in a certain geographical area, which sometimes is only a single village, or other times a small number of villages). But inside that area, they are frequently spoken. And you might be surprised how many of those small-geographic-area languages still have distinct dialects.

For example: my wife (a linguist) did her master's thesis on the pronunciation of a language with about 7,000 speakers, and identified how many distinct dialects there were. (Which is why I know a little bit about this). She recorded native speakers from all 13 (I think it was 13, but it might have been 14) villages where the language was spoken, and found five different dialects, which she grouped into two "main" dialects. (Think American vs British in the English language, with subdivisions into Midwest, New York, and New England accents and so on, and you'll have the right general idea — though these dialects were closer to each other in sound than Midwest vs New York). I'd have to go reread her thesis to give you any more details. But this was a language that was only spoken in a small geographic area, but it was frequently spoken, because that was the main language of those villages. (The country's official national language is what the kids learned in school, but some of the people, mostly those 60 years old or older, hadn't gone to school, because the first government school in their area was only built 60 years ago -- so they only spoke their minority language, and not the country's language, and their kids had to translate for them if they had to leave their village and go shopping in a major town).

[verst]

I would love such a model. My wife's family is Iu Mien which is a sub group of the Dao/Yao Chinese ethnic minority. Mien is its own language but most speakers are essentially illiterate. I'm good with language but there simply isn't a course or any books for learning the language. Not much in writing to begin with given the high illiteracy rate. I would love to build a translation system - project Hail Mary style :)

[yorwba]

FWIW someone wrote their PhD thesis on the Iu Mien language: https://opal.latrobe.edu.au/articles/thesis/An_Iu_Mien_gramm... It's not a textbook, but that doesn't mean you can't try to use it as a textbook. For example, there are some thoroughly-analyzed example texts. Also, the acknowledgments section mentions the Iu Mien Literacy Project, which has published a short course on the very basics, as it turns out: https://iumienliteracyprojects.com/resources/ I suspect the 925-page PDF contains some more hints to additional resources like that. :)

[verst]

Unfortunately there are two sub groups of Iu Mien with somewhat different vocabulary as well - and my wife is from the other group. I had found this pdf document previously - thanks for the pointer! I'll see what I can do with it.

[akreal]

This field got reactivated recently, some new models were released (in the order of release, but the last one seems the best at the moment):

1. ZIPA https://github.com/lingjzhu/zipa

2. POWSM https://huggingface.co/espnet/powsm

3. PhoneticXEUS https://github.com/changelinglab/PhoneticXeus

I would be curious to know how to help people to use these models, or what kind of tasks they could be applied to.

[KingMob]

I learned IPA for Thai, and as part of that, I also read that a lot of professional linguists still find IPA too limited.

IPA seems very comprehensive from my amateur perspective, but apparently a lot of modern linguists still extend it or roll their own.

[sipjca]

Largely this is out of scope for the library, mainly because I’m not aware of many models supporting this. but if there are models which support this would be happy to support

[simsla]

Automatic Phoneme Recognition (APR). There are some models that do this, but they're only so-so.

[msm_]

It sounds like the only way this would make sense is if such model knew the range of sounds it expects to "hear". There's a lot of possible sounds that IPA knows about, but world languages only use a fraction of them at once. Think English dark and light "l" (ball/light) or aspirated "p" (pin/spin) - some languages contrast them, while in english the difference is not meaningful.

Or maybe linguists are actually interested in having maximally faithful IPA representation and manually normalizing it? You are clearly way more knowledgeable about that topic than I, so I'm curious what you think.

[rmunn]

The linguists I know are not necessarily a representative sample... but they're mostly interested in just that: maximally faithful IPA representation. They want to know if the speaker switches back and forth between aspirated p and unaspirated p on the same word, because that tells them something about the language — that aspirated consonants are not meaningful.

Linguists studying the sounds of a language, its phonology, often want to find "minimal pairs", words that differ by only a single sound. For example, din and tin in English. You record a native speaker saying both words, and telling you their meaning, and then you play back either recording A or recording B to other native speakers and ask them which word it is. If they can identify the word every time, then you've found two sounds that are meaningfully distinct in this language. (Some languages don't distinguish the d and t sounds, but English does). But if the native speakers go 50/50 on which word it is, or ask to hear it in a sentence for clarification because it could be two or three different words, then you've found a pair of sounds that this language does not distinguish. (Note that you're playing the words in isolation, because sentence context might make it obvious which one it is, e.g. you can't tell if an English speaker is saying their or there until you hear more words of the sentence).

So yes, the linguists that I know (who, again, are not necessarily a representative sample) are interested in as faithful an IPA representation as they can get, because that inconsistent transcription will give them many clues about the language. It still all has to be checked, because that switching back and forth between aspirated and unaspirated p (for example) could have been an artifact of a poor-quality microphone not picking up the aspiration, or a windy day causing aspiration sounds that the speaker never said, or the speech-to-text model making a mistake. But I watched my wife listen to the same two-second recording on loop over and over, trying to be certain of which sound she was hearing in the middle of the word. Double-checking the output of the model would (in most cases) only require listening to the audio once or twice, not half-a-dozen times like she typically did while researching her thesis. At the time, LLMs were not really a thing yet, but if she were doing her thesis today I bet a speech-to-IPA model would have saved her quite a lot of time — but only if it output every distinction, even the ones not meaningful in the target language. The "maximally faithful" representation, as you put it.

[joshspankit]

Through all this I kept wondering (and excuse the question if it is inappropriate): what’s the benefit of a transcription of the sounds since the ideal is a 1:1 IPA representation to audio recording and neither is itself a translation?

Isn’t the recording itself “more ideal” than any representation of it?

[rmunn]

I don't mind at all.

The main benefit that I see (not being a linguist myself and seeing it second-hand, as it were) is saving time; it's quite a lot of effort for humans to listen to an audio recording and write it down in IPA. I didn't realize just how much effort was involved until I watched my wife researching her thesis, listening to the same recording over and over and over.

Also, there are possibilities for automated translation that the IPA transcription opens up. Speech-to-text models need different training than text-to-text translation models. The other day one of the linguists I know sent me a message, saying "I have a lot of recordings in language X" (I forget which language, but it doesn't really matter which language it is) "and I want to transcribe them. Is there a way to use AI to do that?" I thought it through, and realized that recording-to-IPA followed by IPA-to-correct-alphabet would be a way of dealing with the combinatorial explosion (according to https://www.ethnologue.com/insights/how-many-languages/ there are close to 7,200 languages spoken in the world, though "spoken" may be a misnomer since that number includes sign languages). There's no way anyone can train a model to convert about 7,000 languages from spoken form to written form; that's just way too many possibilities, and I suspect that audio models are necessarily large because audio is more complex information than written text. (If I'm wrong on that, someone please inform me). But although the human mouth (and throat) can produce dozens and dozens of different sounds, that's a finite number. (I think it approaches a couple hundred, but that's still a lot less than 7,000). So the number of distinct sounds and their IPA equivalents that the model would have to distinguish is smallish. Then another model could be trained on converting IPA to the alphabet (and spelling) of the world's 7,000-ish languages — and that would be a lot simpler than sound-to-spelling.

You're right that the recording is "more ideal" than any representation, and linguists will always archive the recording (in a lossless format, whenever possible, though if the original recording was done by a non-linguist who recorded it in MP3 then there's not much point in storing a WAV) alongside their transcription. This allows anyone else who might be doubtful on a point to go listen to the original recording and say "Yeah, Smith got the transcription wrong on that word. That's not an alveolar click, it's a velar click." But when you're writing a paper, you don't stick audio recordings into your document, you write down transcriptions. An audio recording without a transcription is not nearly as useful to linguists as having both. (EDIT to add: And such transcriptions, when you're writing papers about the sounds of a language, are pretty much always in IPA as well as in the language's standard alphabet. IPA isn't just a means to an end, it's also an end in itself. So even if a speech-to-text model existed for the language my linguist friend wanted to transcribe, he would still also want an IPA transcription as well, 90% of the time.)