I wish I could use the Swift compiler from the command line instead of having to download a 10GB XCode monster over my shitty 14kb modem that will take a couple of years.
It's part of the Command Line Tools, which is about 160 Mb on my computer. If you install Homebrew you should get a prompt to install the tools (or try to run gcc from the command line).
If I'm not mistaken, you can use the installation guidelines on swift.org, and just skip the Xcode related parts. I believe you just need to run the downloadable packages and install them, and they shouldn't require Xcode or anything. Then just add Swift's bin folder to your shell path.