Hacker News new | ask | show | jobs
How to kill an unresponsive SSH session (laszlo.nu)
209 points by oarmstrong 4707 days ago
19 comments

Read the SSH man pages every so often, even if you think you know how to use it. There's a lot of features in there. Don't miss the "AUTHORIZED_KEYS FILE FORMAT" in sshd's man page for the uber-cool "command='command'" options for authorized keys (restricts a given key to just be able to run a certain command, very useful). See also SSH's port forwarding, -D, learn how to use ssh-agent, and "man ssh_config".
the command= syntax is (was?) how gitolite works - I'm not sure what the difference between that and a jail is, though.
They're not really related concepts. A jail is allows you to run arbitrary things in a restricted environment while ssh's command feature just forces a particular command to run no matter what's passed.

If your authorized_keys line for a key has command="blah" on it and you go:

    ssh server ls
the server will, rather than running ls as it normally would, run blah and pass the command you specified in as an environment variable. Then blah can parse it and do whatever it wants with it.

You could use command to establish a jail or virtual environment of some sort in which to run the program specified if you wanted to.

The command= simply changes "what" gets executed when the user logs in.

A jail (ChrootDirectory in ssh) changes "where" the user gets when logs in.

The most creative thing I've do with command= was a "select" menu in bash (with some actions in the shudoers).

Other interesting tool I've discovered recently is rrsync. I'm doing the backups of my systems isolated with this. It's distributed with the rsync sources, you put it like:

    command="rrsync /path/to/chroot/the/remote/rsync/client/"
I used to use rsnapshot with command= set to a custom script that would let a few things through (rsync being one of them), by instrospecting `$SSH_ORIGINAL_COMMAND`.
A true BSD chroot jail restricts the user's mobility around the file system, effectively limiting them to running only what binaries have been explicitly placed within that jail. It means, among other things, that the fork hacks that command= is vulnerable to (e.g., vim running other programs) don't work because non-permitted programs basically don't exist as far as the user can see. The downside to jails is that, without automation, they're tedious to maintain, and disk overhead is not insignificant as you have to have a duplicate set of system binaries for each user. In practice, chroot jails end up being very similar to OpenVZ “virtualization” on Linux.
> The downside to jails is that, without automation, they're tedious to maintain, and disk overhead is not insignificant as you have to have a duplicate set of system binaries for each user

On FreeBSD it's common to use nullfs to reduce this - you make one jail skeleton hierarchy and then null-mount it for individual jails so they all share the same underlying files (and cache). zfs clones can be used similarly. I'm sure other OS's have similar capabilities.

That's a lot like the layered model used by docker and warden.
They work in totally different spheres. The command directive can assign different commands per ssh key. Think git@example.com for all of your users, a la Github.

Jails limit the capabilities of users of the OS, instead of users of the network.

Or use mosh[1] on top of SSH and stop worrying about that stuff.

It works much better over high-latency links (mobile). It is not bothered by saturated links, tolerates IP changes and losing the underlying connection like when you suspend your laptop and take it elsewhere.

I now have mosh connect to several servers in tabs when I run gnome-terminal the first time, and only disconnect on reboot. I also run a mosh-capable Irssi Connectbot fork on the phone[2].

It's a massive improvement, fixing many of the little annoyances of ssh.

[1] http://mosh.mit.edu/

[2] http://dan.drown.org/android/mosh/

The server I connect to most often is behind NAT and I'm usually also behind NAT, and mosh doesn't support that config, alas. I'd use it in a heartbeat if I could, though.

https://github.com/keithw/mosh/issues/48

I love mosh, but do note it's a lot slower than plain ssh. I run everything through a 266x188 tmux session, and mosh badly struggles to keep up even with light use. Resizing panes is juddery, vim is visibly laggy, and if anything spams a lot of output to a term (e.g. find /) it basically grinds to a halt. ssh is butter smooth in comparison.

No mouse support in mosh either.

(gnome-terminal/vte (so also Terminator, Xfce Terminal and others) is similarly slow, but as with mosh you don't really notice with smaller terms. I switched to urxvt).

Has anyone audited mosh's security model/encryption? Just curious since ssh is pretty tried and tested.
Authentication and initial key exchange is via ssh, so nothing to audit here. Afterwards data is sent AES-encrypted, which relatively simple.
> Afterwards data is sent AES-encrypted, which relatively simple.

It's simple to encrypt using AES, but that doesn't mean it is simple to encrypt in a secure way (i'm looking at you ECB mode). There are too many ways to accidentally mess up.

It uses ssh to set up the connection, so that setup is fine. The actual traffic is encrypted using AES-128-OCB, which is a great mode of operation.

I haven't digged too deeply into their security, but from what the main page says, it seems pretty trustworthy. Hopefully they're using the appropriate authenticated data, with packet numbers and so forth, to prevent silly things like replay and reordering attacks.

It might have faulty implementation for how it glues security features together, or encrypts the traffic data. I think the parent comment is talking about this kind of "audit" (i.e., not theoretically, but in action and by perusing the source code).
Yes, I addressed the "glue" thing with the note about authenticating the correct data (notably things like IVs and packet numbers).

But given that it uses ssh for the initial setup, I doubt there are any problems to be found with the initial key exchange, which is one of the trickiest parts of a "secure channel" protocol. And OCB, as an AEAD mode, is pretty hard to screw up, so long as (again) you make sure you're authenticating the proper data and generating IVs/nonces appropriately. Plus, they're using the (optimized) reference implementation for OCB, so that's probably pretty solid.

Now, I definitely agree that this isn't a proper security audit. But at a cursory glance, the general impression I get is that it's probably okay. Honestly, they've really removed most of the protocol complexity by using ssh to do all of the setup work.

It still lacks ssh agent forwarding.
…and performs scrollback very poorly.

I still use it, though. Can't wait for them to iron out these issues.

The default escape character ~ does not work if the tilde key in your keyboard layout is a dead key [1], like it is in many European layouts. It can be changed via the EscapeChar config option or the -e command line parameter. It seems, though, that not just any old character is accepted - I tried to use §, which, in the Finnish layout, is in the same physical position as ~ is in the US version, but ssh complains about "bad escape character".

EDIT - I suppose it must be an ASCII character, which is not an entirely unreasonable requirement.

[1] http://en.wikipedia.org/wiki/Dead_key

It works, you'll just have to type

  <tilde> <space> .
Weird - I did try that before writing the comment and it didn't work. May depend on the terminal or something.
I use a french mac layout and it works well. But you have to type ~ then press space, then press shift+; (because shift+; is .)
This must be why mosh's escape sequence never worked for me (ctrl-^ or ctrl-^.)
Seems to be fixed (to use the same as SSH): https://github.com/keithw/mosh/issues/215
You can change what the default escape character is in the ssh config file.
enter-tilde-dot

It is useful, yes. Here's another thing I picked up last week - how do you reboot a remote linux box that's somehow lost its root drive but you still have a shell open (because you left ssh running on another machine)?

  echo 1 > /proc/sys/kernel/sysrq
  echo b > /proc/sysrq-trigger
Another useful trick to remember if you're using Putty and you ever accidentally hit Ctrl-S and find that you've frozen the terminal.

Just type Ctrl-Q and you will unfreeze the connection.

Credit due to: http://raamdev.com/2007/recovering-from-ctrls-in-putty/

If you frequently hit Ctrl-S by accident, and don't consider this a feature, you can turn off XON/XOFF flow control using

   $ stty -ixon -ixoff
then Ctrl-S/Ctrl-Q will be normal characters without side-effects in your terminal, you'll probably just see (depending on shell, terminal, ...) ^S on your screen when you hit it.

To turn it on again, use

   $ stty ixon ixoff
I feel like I should have learned this years ago.
This is not news to anyone (in tech) over 40.

https://en.wikipedia.org/wiki/Software_flow_control

But the world is full of neophytes, re-inventors, etc. It's good to remind people periodically and educate the next generation.
And besides, the older I get (41) the more I can use frequent reminders of the things I used to know. It's disturbing the number of times I've searched the web for answers, and discovered an answer I wrote myself. And I hadn't known this could be easily turned off with 'stty' as suggested in another answer. Or maybe I did once know that?
The following quite becomes more relevant every year: "I've forgotten more than you'll ever know!" --Insulting Old Fogey
Of course. I was actually just expressing a feeling of nostalgia.
That wiki page doesn't mention tilde does it?

I know tilde escapes from cu(1) (part of uucp) http://www.delorie.com/gnu/docs/uucp/cu.1.html

But it may well have originated before then.

i'm turning 30 this year, have been dealing with this issue for 15 years, and still mini-panic for about 2 seconds whenever i hit ctrl-s by accident. it's good to read about.

chill out man.

Being a windows developer who dabbles in linux I accidentally smack ctrl+s a dozen times a day when vimming. I've only just recently got used to to recovery mechanism: Ctrl+q to recover the terminal, a stiff drink to recover my nerves.
From a comment above, I think you'll be able to put stty -ixon -ixoff in your .profile (or .bashrc or whatever) and ctrl+s/q won't be a problem anymore.
I think the implication was that there was a time when one frequently wanted to mess with the flow control.
WELL HOW ESL&^H^HLSE ARE YOU GOING TO READ A 25+ LINE FILE OTHER THAN MASHING CTRLQ BEFORE THE LINES SCROLL OFF THE SCREEN? BY READING THRE AFTERGLOW? I SUPPOSE YOU`VE ASL^H^HSLO GOT A DIRECT CONNECT MODEMS WHERE YOU CAN`T EVEN HEAR WHAT`S GOING ON. +++ATH...
i used to use it all the time on dialup connections. these days it's largely just a pain in the ass.
Thanks for that. In over 20 years I've never worked that one out :)
Thanks - much appreciated!

Despite being brought up on VT220's whacked in a VAX/Sun3 I never got my head around the details.

<enter>~Ctrl-Z will suspend the ssh session, too.

I've also found it useful to do <enter>~C - then you can configure port forwarding without having to open a new ssh session.

(~C opens a command line, enter "help" for available commands.)

Please don't let HN become a substitute for RTFM. This should be known by all SSH users who have skimmed the man page. Fair enough as a blog post but for this trivia to get 46 points so far is deeply depressing.

Maybe I should write a blog post about the use of CTRL-Z in the shell and post that here, should get me Kilo-karma points if this is anything to go by.

When you have a toolset that includes, but is not limited to:

* Ruby

* Rails

* Postgres

* C

* Erlang

* Emacs

* Bash/Zsh/whatever

* Linux Kernel

* GNU C library

* Postfix

and on and on and on, these kinds of tricks are bound to be useful to someone who is not a complete expert with whichever system is being discussed.

Also, realistically speaking, they are far less prevalent than nakedly political articles lately, which do weight on the quality of the site.

This is like not knowing the return or exit function, and is also used for telnet or serial console connections. If knowledge is so shallow on a particular tool, it should not count as being part of the toolset. There's nothing more basic or essential to SSH than connecting and disconnecting. And it's right there on the man page, not exactly a hidden feature. This is more beginner level knowledge, maybe novice if being very very generous, but nowhere near expert level.
Connecting: type ssh and the hostname. Disconnecting: type exit or ctrl-d or whatever on the remote end. That's enough for many people.

For a group of people as large as HN, there are bound to be many people who are experts in one thing, and marginal with others, so articles like this are likely to appeal to them, and be useful.

I would agree if it were for some more obscure yet useful feature, but this is one of the basics, that's clearly documented. This should not be useful for anyone who's ever read the man page, and anyone who's used ssh should have read the man page at least once. That so many people apparently haven't, in what's supposed to be highly technical community, is a very disappointing sign of intellectual indolence.
On my system, I ran this:

    nice man -l -Tdvi man*/* | wc 
And got

3416771 6700199 122007116

At more than 6 million words, that's approximately 10 War and Peace's. And of course in terms of documentation, that's only man pages, which doesn't cover all the stuff that's in other formats like info or html.

You cannot be an expert in everything: I knew about that command for closing ssh, but frankly it is not something I use because it's easier to just close the rxvt and be done with it. I have zillions of other things to occupy my brain.

I'd also argue that in terms of ssh, since this function is so easily accomplished in other ways, this is really just handy triva. Much more useful to know about are all the tunneling things, as they are not necessarily obvious, and can be extremely useful.

I've been using SSH for fifteen years. I have never read the man page because I have never needed to--I have scanned it for specific topics but I have never been quite so teeth-pullingly. This is new and useful information to me and so I upvoted it. On the other hand, I am certain that many, many things I consider trivial would be out of your realm of knowledge and relative understanding of what is and is not trivial should lead you to act better than you are acting.

Or, put another way: don't be an asshole.

At the end of the day, RTFM is much more helpful than spoon feeding someone an answer. Maybe it comes off as being an asshole, but a little shame can be a good motivator for a neophyte to learn how to teach themselves, and they will be better off for it in the long run. "Teach a man to fish" and all that.
Do you want a shovel for that giant hole you're digging?

"A little shame is a good thing" says to everybody around you that you're more interested in feeling good about yourself than helping people. And it's a joke to boot. I've been doing this for around fifteen years, I've written multiple telnet server applications and a MUD client, and I didn't know about this feature in telnet/rlogin/ssh.

Not everyone on HN is a terminal expert, and not sure why you would assume such. It got a lot of points because people found this interesting and helpful. Also, you should write a blog post about CTRL-Z, if people find the info valuable, it'll get the upvotes.
It should also be known by anyone who has used Telnet, since that's where it originated. (For more shortcuts, use '~?' from the ssh session)
Not any telnet I've ever used. Telnet is ^] to get to the command prompt and then "quit". It even says so right when you connect:

    $ telnet google.com 80
    Trying 74.125.239.103...
    Connected to google.com.
    Escape character is '^]'.
Apparently it's rsh, not telnet. The telnet manual does mention it though, since you can put telnet in a rsh-like mode where ~ does work. (May depend on telnet version.)
Ah, my bad! Guess i'm older than I thought...
man 1 telnet on FreeBSD or OpenBSD contains: "When in rlogin mode, a line of the form ~. disconnects from the remote host; ~ is the telnet escape character. Similarly, the line ~^Z suspends the telnet session. The line ~^] escapes to the normal telnet escape prompt."

although on the Debian man page, it appears slightly differently as: "-r Emulate rlogin(1). In this mode, the default escape character is a tilde. Also, the interpretation of the escape character is changed: an escape character followed by a dot causes telnet to disconnect from the remote host. A ^Z instead of a dot suspends telnet, and a ^] (the default telnet escape character) generates a normal telnet prompt. These codes are accepted only at the beginning of a line. "

telnet is not installed by default on CentOS, but has the same man page as on Debian.

Do so if you wish to. Interested to know what you intend to do with that kilo, or how someone else earning a few hurts your cause.
If it becomes acceptable to use HN for gamified man pages instead of expecting people to do their homework, well, there are more than enough man pages in and out of print to fill HN for years. We don't have downvotes on stories, so the only flow control is flagging and complaining about stuff that isn't timely and isn't useful for most of the audience.
That is ridiculous strawman argument. How would such stories end up in a prominent position on HN unless enough people from the community likes it.

In any case that was not what my comment was about. I was, and still am, more than surprised by the poster's puerile lament that someone is getting a few odd karma that the poster thought was undeserved. I assumed that this place is somewhat removed from such "Mama ! he took my toy" behavior.

If you happen to be a few sessions deep, ~~ will send ~ to the next session along. A casual ~~~~~~~~~~~. or so later and everything is wonderful again!
Though you have to be nested really deeply to do ~~~~~~~~~~~. - unlike some other escaping schemes, it only requires one extra tilde for every layer of nesting rather than doubling each time.
The worst problem is trying to figure out just how many layers deep you are. I was being a little ridiculous though.
"So, a totem. It's a small object, potentially heavy, something you can have on you all the time..."
I wonder... I reckon you could implement a totem inside GNU screen that kept track of how many ssh-sessions deep you were.
Not to be confused with: https://projects.gnome.org/totem/
The Telnet Song from CACM, April 1984: http://delivery.acm.org/10.1145/1040000/1035691/p347-steele.... Plain text at http://alcor.concordia.ca/~smw/home/telnet_song.html and no doubt elsewhere.

(This does not refer to the Unix version of telnet, which defaults to ^] as the default escape character.)

Thanks for the tip! I don't tend to nest sessions more than two deep but this could be useful for those who have deep nestings.
In case anyone is having difficulty with the font used on the page, the escape sequence is: newline followed by tilde (~) and then period (.).
The thing that amazes me about this is that people don't realise that this comes from BSD 4.2 rsh released in 1983.
Why does that amaze you? That is a pretty trivial piece of knowledge. Interesting, though.
I'm a little surprised this is on HN? To me this is the equivalent of a blog post about using %d with printf.

Not complaining, just a little surprised something so novice would get attention...

You should post it if you think it will gain traction. Not everyone is experienced with C.
Another great solution is to just use the ServerAliveInterval option.
I can never decide how I want to set this. Most sites recommend 180 seconds or so, but with the default ServerAliveCountMax set to 3, this is 9 minutes before a dead terminal is actually disconnected.

I've started to set it really really low personally - like, 5 seconds, so the connection drops after 15 seconds. I'm tempted to go down to 1, but I have a lot of long running sessions and I start to worry about the traffic counts.

But, after all this there's still a problem: none of it seems to work with any of the connection mux'ing options - once the background session dies, I still have to manually kill it to get anything working again.

While I've frequently used this to kill connections, my favorite thing I've done with it is to list existing and dynamically add new forwarding ports through SSH.
Hitting '.' at a prompt used to be a common idiom for exiting a program. I first saw it when I was a kid working on an HP-3000 system where the system programming language was BASIC and all programs followed this convention. Don't know where it came from originally, but you can still see it in places like rsh/SSH etc.
And SMTP where a single "." followed by CR ends the session.
Does it work too with ctrl-\ ? This has been my process killer recently and it's powerful (and the only way I know to get out of xtail.)
On a side note, does anyone know what ~B actually does? Does it send a SIGINT to the remote terminal? What does ssh mean by the phrase "send a BREAK to a remote system"?

I've tried to use it without success to kill a runaway listing of megabytes of scrolling text, but frantically hitting ctrl-C seems to work much better.

In the beginning, there was RS232, where each character was a fixed number of bits, optionally with a stop bit and a parity bit. If the sender transmitted too many 0 bits to represent a valid character, that protocol failure was called a 'break', and some receiving equipment would detect such a failure and do something about it, like reset itself to a known-good state. Thus, a 'break' was occasionally a useful thing to send, so sending equipment would often have a special keystroke to cause a 'break' condition.

The Unix 'tty' subsystem was basically designed to support simple serial terminals, and so it had a bunch of behaviour designed to interoperate with the pre-existing 'break' conventions. If a Unix system's serial port received a break, Unix would (optionally) send any processes running via that connection a SIGINT, to represent the 'reset to known-good state' behaviour (this is controlled by the stty command's 'brkint' flag). Also, if the user's terminal didn't provide a specific 'send a break signal' command, Unix could be configured to send a break signal when it received some particular character (^C by default; this is controlled by stty's 'intr' setting).

Of course, nobody uses physical RS232 terminals anymore, but for compatibility reasons the Unix tty API lives on, and the "psuedotty" implementation used for things like terminal emulators maintains compatibility. ssh is basically a tool for exporting the tty API over the network, and so for compatibility it too must have a way to transmit the information "pretend a break condition has occurred on the RS232 connection we're pretending to use."

To summarise: yes, unless you've messed with the stty command, ~B will probably result in a SIGINT. Ctrl-C is probably more reliable since you can hit it much faster than you can type <Enter>~B, but ~B is still useful if your terminal is in 'raw' mode, where ^C is not converted to SIGINT (for example, if you're running an app that wants to bind ^C to some other function).

See also:

    https://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter#Break_condition
    stty(1)
    tcsendbreak(3)
This usually doesn't work for me. Perhaps it's because my sessions are usually multiplexed via "ControlMaster auto"?
[Enter] Shift+~ .
I'm surprised/apalled at how many people upvote this, considering how this place is supposed to be "hacker news"...
you really don't HAVE to know this to be a good "hacker"

That being said, some people are new to the game. No reason to be elitist.

Sometimes I upvote articles where the comments are good, present new information or just have better information than the article itself.