It seems a pretty obvious portmanteau of magic + git. I always said it with a different rhythm: "ma-git" rather than "mag-it", so that might also be a factor.
I always pronounced the G softly as in 'magic' so it took a while for the criticism to register. Gonna be honest, I like magit with a hard G, maggots 'n all.