Age is a factor though in how people think and perceive the world yeah? I do apologize thankfully I'm not in a position where my bias hurts anyone but myself and I'll work on my thinking.
Let's work from a definition here; I think this is the kind of "ageism" we're talking about (sourced from dictionary.com):
> a tendency to regard older persons as debilitated, unworthy of attention, or unsuitable for employment.
I think this clarifies things, doesn't it? Even if you somehow objectively prove that older workers are unsuitable, or less suitable, for employment, that doesn't make a philosophy against hiring older workers not-ageist. It just means that ageism is correct (to be clear: hypothetically. I am not endorsing that view).