[woah]

On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

[mendeza]

I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation.

Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reali...

http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T...

And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText

[tasogare]

How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.

[TrainedMonkey]

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

[echelon]

Humans can transfer learn across domains because we can draw on an incredible wealth of past experience. We can understand and abstractly reason about the architecture of problem landscapes and map our understanding into new spaces.

That isn't even counting our hardwired animal intelligence.

[w4n9r3ntru3]

Humans learn efficiently, but that has nothing to do with having a lifetime of data I believe. Humans have a lifetime of data, but you can easily parallelize the model such that it takes in more data than a human can in his/her lifetime, and still humans are still the state of the art.

[cygaril]

Or our entire evolutionary history of data.

[AlanSE]

...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.

[eanzenberg]

I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon

[K0SM0S]

It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us.

The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself.

It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).

[tprice7]

"The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself."

Yes, and they do. They aren't choosing completely arbitrary algorithms when they attempt to solve a ML problem, they are typically using approaches that have already been proven to work well on related problems, or at least are variants of proven approaches.

[K0SM0S]

Indeed, but what's the magnitude?

The question is, how much information is encoded in those algos (to me, low-order logical truths about a few elementary variables, low degree of freedom for the system overall), compared to how much information is encoded in the "algos of the human brain" (and actually the whole body, if we admit that intelligence has little motivation to emerge if there's no signal to process and no action to ever be taken).

I was merely pointing out this outstanding asymmetry, as I see it, and the unfairness of judging our AI progress (or setting goals for it) relatively to anything even remotely close to evolved species, in terms of end-result behavior, emergent high-level observations.

Think of it this way: a tiny neural net (equivalent to the brain of what, not even an insect?) "generationally evolved" enough by us to be able to recognize cats and license numbers and process human speech and suggest songs and whatnot is really not too shabby. I'd call it monumental successs to be able to focus a NN so well on a vertical skill. But that's also low-order low-freedom, in the grander scheme of things, and "focus" (verticality) is just one aspect of intelligence (e.g. the raging battle is for "context" as we speak, horizontality and sequentiality of knowledge; and you can see how the concept of "awareness", even just mechanical, lies behind that). So, many more steps to go. So vastly much more to encode in our models before they're able to take a lesson in one standing and a few examples.

It really took big-big-big data for evolution to do it, anyway, and we're speeding that up thanks to focus in design, and electronics to hasten information processing, but not fundamentally changing the law of neural evolution, it seems.

If you ask me, the next step is to encode structural information in the neuron itself, as a machine or even network thereof, because that's how biology does it (the "dumb" logic gate transistor model is definitely wrong on all accounts, too simplistic). Seems like the next obvious move, architecturally.

[eanzenberg]

humans dont start with a random scrambled brain

[lallysingh]

Is that in a csv.gz I can torrent somewhere?

[rhizome]

Are you referring to empiricism?

[ummonk]

Transfer learning for the win.

[SQueeeeeL]

Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make.

Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)

[ssivark]

Agree with your first statement and disagree with your second; I don’t think the former implies the latter.

I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.

[btrettel]

These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables.

As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.

[gdubs]

This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.