I think I'd prefer to have all ignores and un-ignores explicitly in the file and not have some of them defined implicitly because a file was added to tracking at some point.
But ignore files are only for untracked files anyway. Maybe you want them to specify what can be in the repo and what not, but this is not how Git works.