I am with you. That said, I feel like looking at a photo when I was very young say around 4 or 5 does seem to invoke memories about the house I grew up in. I wonder if the brain is just making up the memory or the photo is actually helping the brain access some dormant region in the brain.
Further, I have also noticed that when I meditate sometimes my mind wanders and brings me some memories from the childhood - totally random, unrelated to current events memories.
I have a decent memory for places, so-so for people, very little for events. I could draw a reasonable floor plan of my childhood home and tell you about friends and family who were there. But I could only tell you about a handful of things that happened in that place.
It's not much better in more recent times. I could say much the same about everywhere I've lived as an adult.
Further, I have also noticed that when I meditate sometimes my mind wanders and brings me some memories from the childhood - totally random, unrelated to current events memories.