I've only been on HN since around ~2009 or something like that (11 years), but most people I've met in real-life who are also on HN have been recommended to visit the site first time from fellow hackers, rather than via other websites.
I was a reader of Paul Graham's essays and website, which I believe is where I heard about it. Or it might have been reddit (which I'm sure I heard about via PG). I joined Reddit on January 25, 2006 and HN on February 21, 2007.