[vmurthy]

>One of the things he told me, and that I also observed, was that the vast majority of financial experts (basically, the people in the accounting department of companies) had an extremely difficult time just telling him what the rules of any particular transaction should be

We have internalized more knowledge than we can explain sounds like the textbook definition of Polanyi's paradox:"Polanyi's paradox, named in honour of the British-Hungarian philosopher Michael Polanyi, is the theory that human knowledge of how the world functions and of our own capability are, to a large extent, beyond our explicit understanding" [0]

[0] https://en.wikipedia.org/wiki/Polanyi%27s_paradox

[hn_throwaway_99]

Thanks so much! I hadn't heard of Polanyi's paradox before but it is exactly what I was talking about. The Wikipedia entry even highlights the problem for AI-driven development:

> Polanyi's paradox has been widely considered to identify a major obstacle in the fields of AI and automation, since programming an automated task or system is difficult unless a complete and fully specific description of the procedure is available.

[rcxdude]

I'd say the opposite in some ways: it's arguably the reason for the bitter lesson and ML techniques tending to beat other approaches, because they work by giving examples as opposed to trying to articulate the rules in an explicit manner.

[formerly_proven]

cf https://en.wikipedia.org/wiki/Dual_process_theory as a causative agent behind the paradox.

[reemyoass]

having worked on automation of accounting process, it has been a combination of ever evolving ruleset/edge scenarios for 10% cases plus standardised rules for 90% of the cases. i think the problem is that it is an evolving field. alot of times it borders law, which makes it murky and vague too.

[taneq]

This reminds me of that quote about how if our brains were simple enough for us to understand, we'd be too simple to understand them.

[halfcat]

The body can know more than it can tell.