Well Chrome does have an installation identifier that was/is transmitted only to google properties.
That would explain the behavior you describe. Google says it is only for tracking experiments and tests, but I doubt even they know what everybody is doing with it, and perhaps you were a member of an experiment where they used it to track you.
"Google tracks individual users per Chrome installation ID" [0]
I use firefox and I see the same behavior with web&app activity turned off, and youtube history turned off. Personalized frontpage when I visit youtube normally. If I open incognito then I get unpersonalized. If I then login inside the incognito session, then I get the personalized recommendations again. This confirms that the recommendations are tied to the account.