[avianlyric]

Based on my understanding of the article, it sounds like OP has re-discovered/invented Knowledge Distillation[0]. Which is reasonably well research technique, and frequently applied to get models to run on compute constrained devices. Or taking very deep models, and using them to train very shallow, but low-latency models.

[0] https://en.m.wikipedia.org/wiki/Knowledge_distillation

[Buoy]

Glad to know that's what it's called! For avoidance of any doubt I'm not trying to claim I 'discovered' any of this - just wanted to share some useful learnings :D

[TechBro8615]

When you independently discover commonly used techniques (often without even realizing it), it's usually a good sign you're working on and thinking about cool stuff! Eventually one of those ideas will be entirely novel (you've probably already had many of them), and that's how scientific progress is made.

[crubier]

I came here to say this. This entire article should have been 5 lines, had they known about Knowledge Distillation.

Anecdotally, a similar thing happened to me when I "independently invented" synthetic data generation for deep learning back in 2016... Turns out my brilliant idea was not that original after all

[TechBro8615]

The more I learn, the more I realize how rarely I've thought of a "new idea." In fact, I'm not convinced such a thing even exists. An idea is generally a set of connections between pre-existing ideas, so perhaps a novel subset of those connections would qualify as a "new idea," but one could argue it's more of a rearrangement of existing concepts. (What was the first idea? The first thought? Does thought require language, and if so, is consciousness an emergent property of language?)

When I realized how rare it is to have a unique thought, I reframed my approach to evaluating my own ideas. Instead of worrying about novelty, I worry about relevance.

If I can't find a pre-existing similar idea, I assume I'm on the wrong track unless I can identify a compelling reason why I'm in a unique position to have an idea that nobody has ever thought or at least felt like communicating publicly.

If I can find such a pre-existing idea, then I know I'm on the right track, and often it validates a pattern I identified between a subset of connections in my previous knowledgebase, solidifying the pattern in my mind. Regardless of whether I can label it a "new idea," I can use it to work toward the next step in whatever line of thinking caused me to notice the pattern in the first place.