Yep, I agree and that's where I install stuff on everything other than OS X. One distinction though, I think, is that most of the stuff I install on OS X I don't want to be available system-wide. I'm (typically) not using Homebrew to install daemons that run all the time or things that serve critical system/network functions, so I've never seen a reason to make them available to the entire system. I agree that goes against the Unix way but I started preferring this way of using Homebrew after I had similar problems upgrading and even updating OS X.
Also, what if for some reason a single machine is shared by two people and they need different versions of some programs installed with Homebrew? Installing everything in /usr/local isn't going to look like a good idea then.
It's not going to look like a good idea even if they need the same version. You're not supposed to run homebrew as root; it runs as your own user instead. If two users try to install things with homebrew in the same directory, you're going to end up with some things owned by one user and some things owned by the other, and things will start failing pretty soon.
A while ago I floated the idea of having a separate, low-privilege "brew" user that installs things, with the brew command automatically switching to that user, but there was no interest.
It is if by manually managed we mean _not managed by the system/OS, and not updated without user interaction_. If you update OSX and you had homebrew binaries in /usr/bin, they'd be gone. Homebrew also won't you that there are updates and never updates packages unless you decide to get a new version.
Also, what if for some reason a single machine is shared by two people and they need different versions of some programs installed with Homebrew? Installing everything in /usr/local isn't going to look like a good idea then.