I usually play it on a tty, partly because silly decisions, but also I can ssh into a my system that has it installed. I haven't seen a 'noticeable' slow down, but then again most of my forts barely make it to 80 dwarves.
From my experience a few releases ago, the game is completely playable in text mode, over SSH. Like others have said, it's a little slower. I think some key combinations work differently too.