[schoen]

If you had a 2048-bit public key modulus, each factor (only one factor is sufficient to reconstruct the private key) is only about 308 decimal digits, or 256 hex digits. :-)

We also know from Nadia Heninger and Hovav Shacham's research that you can reconstruct private keys relatively efficiently if you have some missing bits.

https://eprint.iacr.org/2008/510.pdf

But I think you're right that human memory isn't a very significant threat to RSA private parameters. Realistically, cameras would be the threat, not a human being glancing it them.