My impression is that 4chan did a Mother Night on itself. They started with ironic Hitler memes and edgy teenage shit, but eventually people took that seriously and all that were left were edgelords and nazis.
I know it's comforting to believe that there were never any real racists on 4chan until relatively recently, and that it was all naive shitposters and kids making edgy memes, but it's far more likely that actual racists have always hidden behind the pretense of 4chan's ironic culture and anonymity, and have always been active there.
In the olden days you could read between the lines to see people mocking the very culture their post was supposedly glorifying. Later this nuance disappeared as the board was overrun with actual Neo-Nazis.
The folklore was that the neo-nazi forum Stormfront saw an opportunity and started astroturfing legit Nazi views on boards like /new/, causing this transition (and thus changing the course of western politics for years to come).
I never looked deeply into it, but I don't think the transition happened entirely naturally.
People have an impression that 4chan is exclusively /pol/ and /b/, but a lot of it's fantastic.