The Netherlands was the last country in Europe to introduce patent law AFTER Philips stole bulb manufacturing technology from Edison (Philips is now a huge patent holder and actively steals ideas from startups to turn them into patents).
If you can't innovate, buy (or steal) someone else's invention, and use a government granted monopoly (i.e. patent) to prevent anyone else from innovating further and making a better version.
Maybe patents provide an incentive to be innovative, but they also create a barrier to innovating on top of technology that is protected by patents.
Some people remembering things and going elsewhere and using what they remember seems a little different from copying of millions of documents and schematics and plans.
I'd say it is the same difference between a police officer remembering a license plate for the getaway car of a bank robbery, and having pervasive automatic surveillance tracking everywhere everyone goes.
Yes, I guess it is very efficient to not need to spend any money on R&D, and just steal from those who do spend the money.
Will anyone spend money on R&D in this efficient world when the result is you just go out of business because you can't compete against anyone who does?