I usually play it on a tty, partly because silly decisions, but also I can ssh into a my system that has it installed. I haven't seen a 'noticeable' slow down, but then again most of my forts barely make it to 80 dwarves.
From my experience a few releases ago, the game is completely playable in text mode, over SSH. Like others have said, it's a little slower. I think some key combinations work differently too.
I believe it was originally displaying extended ASCII in a nCurses console, and switched to bitmaps with SDL. The majority of the world is still visualized by those character codes and two colors, but entities can be assigned arbitrary sprites http://i.imgur.com/zuMdqbv.jpg
Actually it was the other way around. It was originally a graphical game which deliberately used a set of bit-mapped tiles drawn from the CP437 (not ASCII!) font found on many old computers. It later grew the option to add in extra graphics, so that a dwarf or a cat is a picture of a dwarf or a cat rather than a colorized letter. In this mode it's actually an OpenGL game, although it doesn't do much with the hardware.
A release or two back it gained a true console mode which you can play on a real terminal.
I wouldn't recommend it, because it's noticeably slower than the SDL or OpenGL renderers, but you can.