[wongarsu]

Often you have chains of invention, where it's not obvious from the outset that a specific outcome can be achieved by making that specific discovery or invention.

Other issues are cases where the profit to society is clear, but difficult to capture by any specific group (the reason why infrastructure is mostly built by governments), and cases where discoveries happen by chance because somebody was given the chance to "screw around" (for example the person who found the xz vulnerability probably couldn't have justified why he put engineering time into investigating a 0.5 second delay).