That's because you are visibly American. If you looked more like a southerner, you'd have much better, they wouldn't switch to English. On the other hand, they wouldn't even speak a Norwegian to you.
I don't think it's looks, it's accent. Often someone will start a conversation with me in Norwegian before I've spoken, then switch when I answer. Granted, My Norwegian is not great yet (A2/B1 level), but it still definitely makes it harder to practice and learn here.