Deleting code is not always better. I know a lot of developers who like to "refactor" code by making everything more terse and they regard that as a win even though the code is now cleverly incomprehensible.
I agree, it depends of the code, and if you need to replace it: by what.
I'm actually more found of when I don't even have to replace it: just delete the damn obsolete shit. Not replace. Delete. Most of the time it for cases where it is just dead. Sometimes it is only quasi-dead (from an incomplete previous modification) and in a few cases deleting it actually fixes some bugs.