The dirt in your back yard hasn't been around for 4.4b years. Topsoil (decayed organic material and newly eroded rocks mostly) is usually pretty 'young' on geological timescales.
Right, but at least we can say you're a relatively old "living" creature where age is a more relevant number. (I'm getting up there in "relatively old" years too.)
By the same definition, I'm not 40 years old either, because practically no part of me is left since I was born.