[ScottAaronson]

Results from inductive inference show that you’re not going to converge in general. In the specific case of chess, however, my point was that we, because we know the complete list of rules, could plausibly look in from the outside and say after some finite point: “ah yes, now the system has gotten all of them.”

[YeGoblynQueenne]

Oh, I see. Yes, for sure. That'd be the active learning framework [1]. I get the feeling it's not very popular because it inserts a human in the process. Many machine learning people are rather allergic to anything that a) introduces "human bias" to the learning process and b) reduces the potential for end-to-end automation.

The article you quoted above is a good example. It's main claim is that an evaluation function was learned without knowledge of rules or hand-crafted features, i.e. explicit human participation.

It's a political issue, really. And a silly one- there are domains where very useful background knowledge is available; physics, language, mathematics, etc. There's no reason not to use it.

In chess, for example, it might be possible to learn a decent set of rules just by carefuly choosing the examples to feed to the learner. Or, indeed, evaluating the learning so-far, therefore acting as an all-knowing Oracle.

___________________

[1] Well, strictly speaking active learning is where the system asks the user for examples/ counterexamples, but there's different options and what you suggest would fit in there.