Maybe the time traveler’s grandfather got killed in a temporal suicide attack? Thus preventing the time traveler from being born. Thus why we don’t yet have time travel.