The neurological argument is very much in its infancy, with very little in the way of true experimental evidence, yet laypeople throw it around as if it's well-established theory. Neuroscientists, however, are pretty humble and hedging about it: every paper on the subject is littered with mights, maybes, perhapses, and "more investigation is needed"s.