[tdekken]

Great question!

I seem to be in a similar situation as an experienced software engineer who has jumped into the deep end of ML. It seems most resources either abstract away too much detail or too little. For example, building a toy example that just calls gensim.word2vec doesn't help me transfer that knowledge to other use cases. Yet on the other extreme, most research papers are impenetrable walls of math that obscure the forest for the trees.

Thus far, I would also recommend Andrej Karpathy's Zero to Hero course (https://karpathy.ai/zero-to-hero.html). He assumes a high level of programming knowledge but demystifies the ML side.

--

P.S. If anyone is, by chance, interested in helping chip away at the literacy crisis (e.g., 40% of US 4th graders can't read even at a basic level), I would love to find a collaborator for evaluating the practical application of results from the ML fields of cognitive modeling and machine teaching. These seemingly simple ML models offer powerful insight into the neural basis for learning but are explained in the most obtuse ways.

[ttul]

Jeremy Howard's FastAI's course is another great one: https://course.fast.ai/

I'm enrolled in their latest course via University of Queensland; presently, they're teaching us by implementing one of the latest text-to-image papers in PyTorch. They cover the math as side lectures if you're interested in it and have the pre-requisite knowledge. But it's not necessary if what you're keen on is the programming of models.

[Heidaradar]

great, thanks!

also for your PS, can you give a little more detail? What's your end result, what have you done so far etc

[tdekken]

> what have you done so far

So far, I am a week into learning ML :). I have spent ~30 hours watching various ML courses and am in the process of testing the hypothesis that teaching reading with a shallower orthography (e.g., differentiating between the short and long 'e' sounds by introducing an 'ē' grapheme) leads to improved recognition of sublexical patterns. The step I am working on is building an embedding layer to ensure that these new graphemes (i.e., 'ē', 'ā', etc.) are near their parent grapheme (i.e., 'e', 'a') in the embedding space. (Although the model seems straightforward, I could also be completely misguided in how I am tackling this problem :) ).

FYI, this orthographic approach (i.e., how words are spelled using an alphabet) is used in a few highly researched literacy programs, but AFAICT there isn't direct research on the approach itself. The motivation is to initially make English a consistent language (i.e., the letters you see have a one-to-one correspondence with a particular sound). This should greatly simplify the initial roadblock in learning to read English (as seen by studies of countries with "shallow" orthographic languages) and then learners would transfer this knowledge to the normal (inconsistent) English orthography.

[Heidaradar]

damn, this all sounds like very interesting, cool stuff!! I'm not sure if I'd be able to help much/have the time for it though, but best of luck!

[tdekken]

I would love to!

My main goal is to use cognitive modeling to evaluate the efficacy of interventions and inform the personalized "minimum effective dose" for a particular learner. Academically, this is well-trodden territory [0-2] but these results haven't found there way into practice. This is critically important because we know that ~30% of children will learn to read regardless of method, ~50% require explicit, systematic instruction, ~15% require prolonged explicit and systematic instruction, and up to 6% have severe cognitive impairments that make acquiring reading skills extremely difficult [3]. Yet, how much is enough?

To make this more concrete, imagine you are learning a foreign language with Duolingo. How much effort per day is necessary to achieve that? Many people have long streaks and are no closer to fluency (I learned nearly nothing despite a 400 day streak). Similarly, many reading interventions are once-a-week and, predictably, don't meaningfully affect the learning outcomes for those students.

BTW, this ML portion is part of a much larger effort (e.g., our team is a Phase II finalist in the Learning Engineering Tools Competition). If anyone is interested in collaborating, please feel free to reach out to me.

[0] Phonology, reading acquisition, and dyslexia: insights from connectionist models (https://pubmed.ncbi.nlm.nih.gov/10467896/)

[1] Modeling the successes and failures of interventions for disabled readers. (https://www.researchgate.net/publication/243777699_Modeling_...)

[2] Learning to Read through Machine Teaching (https://arxiv.org/abs/2006.16470)

[3] Education Advisory Board. (2019). Narrowing the Third-grade Reading Gap: Embracing the Science of Reading, District Leadership Forum: Research briefing

[thundergolfer]

Don’t get me wrong I think your work is really cool and a worthy cause, but surely the literacy crisis is a socio-economic problem not a technological one.

[tdekken]

> surely the literacy crisis is a socio-economic problem not a technological one.

Yes and no. It is, of course, not strictly a technological one, but the argument that it is a socio-economic one is, at best, an oversimplification. If you are interested in a more complete understanding, I highly recommend checking out APM's documentaries on this issue (https://features.apmreports.org/reading/).

From my research, the underlying causes of the literacy crisis are:

1. The mistaken belief that reading, like speaking, is biologically natural. This belief manifests as guidance to surround your child with books and read to them. Unfortunately, this isn't sufficient for the majority of children.

2. The majority of teachers lack the content knowledge to teach children to read. For example, imagine helping a child to sound out the word "father". What is the sound of the second letter? It isn't a short 'a' nor a long 'a'.

3. Many popular programs used in schools are completely debunked by science (e.g., cueing theory), but as a teacher it is difficult to identify that your approach is faulty. (If ~30% of children learn regardless of method, it is too easy to offer excuses for why the other children don't learn).

4. Helping a struggling child is "rich man's game". If you are high SES and your child is struggling, you will pay a tutor to rectify the problem. That isn't an option for the vast majority of families.

In other words, this is a highly complex puzzle and it is completely understandable why society is seemingly no closer to solving it :). Consequently, the majority of our effort is directed at understanding these root causes and identifying how to overcome them (FWIW, we have made significant progress here). The cognitive modeling portion is a small but plausibly important part of the larger landscape.

[thundergolfer]

Thanks for the answer. I haven't learned that much about this topic, but most of what I have learned is from reading E.D Hirsch Jr.

Who would you recommend I read next?

[tdekken]

It depends how far down the rabbit hole you want to go :). I highly recommend checking out APM's documentaries on this issue (https://features.apmreports.org/reading/). These are in-depth and accessible.

If you want to go further, you can read Moat's Speech to Print, Seidenberg's Language at the Speed of Sight, and many others. If you want to go even deeper, then welcome to the firehose that is educational research :D.