I don't agree with the connotation, but unfortunately there are negative associations in American culture with older white men who are into anime or other Japanese cultural artifacts.
Sure, there's a certain stigma attached to anyone who enjoys Japanese culture -- older white men included. But even then, one of the words that pop into my mind when thinking about said stigma isn't rape. It's weeaboo, nerd, otaku, geek, parents basement, neckbeard, etc etc etc.... I say that as someone who enjoys japanese culture from time to time myself. Not anime, but j-dramas, movies and their music. Especially their music. It reminds me of how music was here in the 90s (not as manufactured, still a lot of experimenting going on, etc).