[chimi]

It's hard to verbalize this, most of it is "intuition" but I think it boils down to "supervised learning is BS."

Humans are smarter than computers. How can a human teach a computer how to do something when the human itself can't teach another human that something?

We haven't solved that problem. The snake is eating its tail.

You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No".

Well.... why? Why is it yes or why is it no? What is the difference between what the human or the computer, or in general, the student, did and what is good or correct? And then you still have to define "good" and many times that means waiting, in the case of the PDF linked to above, perhaps many years to determine if the employee the AI picked, turned out to be a good employee or not.

And how do you determine that? How do you know if an employee is good or not? We haven't even figured that out yet.

How can we create an AI to pick good employees if human beings don't know how to do that?

Supervised learning isn't going to solve any problem, if that problem isn't solved or perhaps even solvable at all.

In other words, over the years, my heuristic has turned into, "Has a human being solved this problem?" If not, then AI software that claims to is BS.

[tensor]

Supervised learning in machine learning is nothing remotely like a human teaching anyone anything. It's a very clear mathematical formulation of what the objective is and how the algorithm can improve itself against that objective.

The closest analogy for humans would be to define a metric and ask a human to figure out how to maximize that metric. That's something we're often pretty good at doing, often in ways that the person defining the metric didn't actually want us to use.

[chimi]

> Supervised learning in machine learning is nothing remotely like a human teaching anyone anything.

I disagree, I think it's exactly the same. As an example, a human teaching a human how to use an orbital sander to smooth out the rough grain of a piece of wood.

The teacher sees the student bearing down really hard with the sander and hears the RPM's of the sander declining as measured by the frequency of the sound.

The teacher would help the student improve by saying, "Decrease pressure such that you maximize the RPM's of the sander. Let the velocity of the sander do the work, not the pressure from your hand."

That's a good application of supervised learning. Hiring the right candidate for your company is not.

[Thrymr]

But that's not at all how "supervised learning" works. You would do something like have a thousand sanded pieces of wood and columns of attributes of the sanding parameters that were used, and have a human label the wood pieces that meet the spec. Then you solve for the parameters that were likely to generate those acceptable results. ML is brute force compared with the heuristics that human learning can apply. And ML never* gives you results that can be generalized with simple rules.

* excepting some classes of expert systems

[chimi]

One of the columns of sanding parameters is the sound of the sander.

[bildung]

Machine learning really almost nothing in common with most types of human learning. The only type of learning that has similarities is associative learning (think Pavlovs dogs studies).

The human learning situation you describe works quite differently, though: The student sees either the device alone or the teacher using the device to demonstrate its functionality. This is the moment most of the actual learning happens: The student creates internal concepts of the device and its interactions with the surroundings. As a result the student can immediately use the decive more or less correctly. What's left is just some finetuning of parameters like movement vectors, movement speed, applied pressure etc.

If the student would work like ML, it would: hold the device in random ways, like on the cord, the disc, the actual grip. After a bunch of right/wrong responses she would settle on using the grip mostly. Then (or in parallel) the student would try out random surfaces to use the device on: the own hand (wrong), the face of the teacher (wrong), the wall (wrong), the wood (right), the table (wrong) etc. After a bunch of retries she would settle on using the device on the wood mostly.

It's easy to overlook the actual cognitive accomplishments of us humans in menial tasks like this one because most of it happens unconsciously. It's not the "I" that is creating the cognitive concepts.

[scarlatti]

That is such a horrible metaphor

[mark-r]

> You can't teach a human how to do something when the methodology to do that is the student trying something and the teacher saying "Yes" or "No".

Strangely, I recently had to complete a cognitive test that was essentially that process. I was given a series of pages, each of which had a number of shapes and a multiple choice answer. I was told whether I chose the correct answer, then the page was flipped to the next problem. The heuristic for the correct answer was changed at intervals during the test, without any warning from the tester. I'm told I did OK.

[chimi]

You're touching on the "difficulty" in verbalizing it. I see what you mean, because you did learn that the heuristic was changing with just a yes or no. I said you can't teach that way, but you clearly learned that way, so I wasn't exactly correct, but I'm not practically wrong either still I don't think.

I wonder, how would an AI perform on the same test.

What is the mathematical minimum number of questions on such a test, subsequent to the heuristic change, that could guarantee that new heuristic has been learned?

I'm curious about the test. Did it have a name? What were they testing you for?

[visarga]

> I wonder, how would an AI perform on the same test.

This situation is called Multi-armed Bandit. In this setup you have a number of actions at your disposal and need to maximise rewards by selecting the most efficient actions. But the results are stochastic and the player doesn't know which action is best. They need to 'spend' some time trying out various actions but then focus on those that work better. In a variant of this problem, the rewards associated to actions are also changing in time. It's a very well studied problem, a form of simple reinforcement learning.

[rusticpenn]

If the rewards are changing, then isnt it a moving target problem?

[gbear605]

Doesn’t it depend on what you mean by guarantee? The test can’t get 100% certainty, since theoretically you could be flipping a coin each time and miraculously getting it right, for 1000 times in a row. The chance of that is minuscule (1/2^1000), but it’s nonzero. So we’d have to define a cutoff point for guaranteed. The one used generally in many sciences is 1/20 chance (p = 0.05), so that seems like a plausible one, and with that cutoff, I think you’d need five questions passed in a row (1/2^5 = 1/32). Generally, if you want a chance of p, you need log2(1/p) questions in a row passed correctly. However, that only works if your only options are random guessing and having learned the heuristic. If you sorta know the heuristic (eg. right 2/3 of the time), then you’d get the 5 questions right ~13% ((2/3)^5) of the time, which isn’t inside the p = 0.05 range. So you also need to define a range around your heuristic, like knowing it X of the time. Then you’d need log(1/p)/log(1/X) questions. For example, if you wanted to be the same as the heuristic 19/20 times and you wanted to pass the p = 0.05 threshold, you’d need log(1/0.05)/log(1/(19/20)) ~= 59 questions.

[mark-r]

There were more than two possible answers to choose from on each page, so the odds of being right were considerably lower.

[mark-r]

I'm sure the test was a standard with a name, but I was never told. It was a small part of a 3 hour ordeal, evaluating my healing progress since suffering a brain injury in March.

I would agree that it's a very inefficient way of teaching something. It gave me an unexpected insight into machine learning though.

I'm sure the test was designed so that picking the same answer each time or picking one at random would result in a fail.

[munin]

Sounds like Ravens progressive matricies.

[mark-r]

Similar but not the same.

[wilg]

Well... why is it necessary that we can teach a human to do something in order to teach a machine to do it?

[mark-r]

Teaching a human is a heuristic for understanding the problem well enough to teach a machine.

[chimi]

I agree and rather than post a sibling response, I'll add that I think it's necessary today, simply because we don't have AGI, yet. And also point out that we are talking about determining if AI is snake oil or not. There may be some scenarios where we can teach a computer to do something we can't teach a human to do, I can't think of any off the top of my head, but if we can't, then I'm going to be super doubtful that an AI software can do it better than a human, if at all.

AGI, in the singularity sense, will be solving problems before we even identify them as problems. Experts in a field can do this for the layman already and I think it's possible. Some don't. I do.

It'll be super interesting when it flips! When the student becomes the master and we, as a species, start learning from the computer. You can kind of get a sense of this from the Deep Mind founder's presentation on their AI learning how to play the old Atari game Breakout. He says when their engineers watched the computer play the game, it had developed techniques the engineers who wrote the program hadn't even thought of.

Even still, the engineers could teach another human how to play Breakout, so yes, I do believe they did in fact create a software to play Breakout better than they could.

[visarga]

Same for AlphaGo, but it only works when you have access to cheap simulation (breakout being a game easy to run, Go being just a board). It doesn't work as well in situations where you don't see the full state of the system, or where there is randomness.

[xorfish]

AlphaGo did pretty well in Starcraft 2. Even though it is still pretty far from the best players in the world.

[MaxBarraclough]

This simply isn't true. We know your intuition here is mistaken, as we have plenty of counter-examples.

The best chess AIs can beat any human chess player. They use techniques that were never taught to them by a human.

Another example: a machine-learning-driven computer-vision system predicting the sex of a person based on an image of their iris. No human can do this. [0]

[0] Learning to predict gender from iris images (PDF) https://www3.nd.edu/~nchawla/papers/BTAS07.pdf