No, you can build pretty much everything from homebrew using just the command line developer tools. It's been like that for 7-8 years, if not more. Off the top of my head I've only once ever run into a homebrew recipe requiring "xcodebuild" to proceed.