Having been tasked with throwing up a new blog post on our shiny new site, I've decided to devote my first verbose rant to something that's an easily solved annoyance - SSH timeouts. On the projects I end up attached to, I often encounter people who aren't actually systems administrators. Shocking, I know! What tends to happen is sooner or later, someone attempts to execute a long-running command with no intermediate output. Inevitably, their SSH connection is closed due to inactivity, and their command ceases execution, potentially causing hours of delays. Such is the wonderful modern world we live in, a world of NAT, and firewalls, and of load balancers; all working to aggressively close idle connections. Gone are the days when one could telnet into a server and have a connection remain open, with no input or output, for days or weeks at a time.
'Tis a horrible loss, yea, but worth it - because seriously, telnet? How did anyone ever think that was a good idea?
I digress. The best solution for executing long-running commands is a well-known program by the name of 'screen'. In a nutshell, screen allows you to create a session, execute commands, detach from that session and log out - whereupon the session will continue to happily run, waiting for you to get back from plundering the Spanish Main or burying your treasure or whatever it is that's so much more important than staring blankly at a monitor.
The basic knowledge you need is pretty simple...
Start a new screen session:
List existing screen sessions:
Detach from current session:
Reattach to a session:
screen -r <screen session>
Close a screen session:
screen ./longrunningcommand.sh ctrl+a+d (logout as normal) (come back later) screen -ls (locate your screen session) screen -r YOURSCREENSESSION ??? Profit.
It's that simple.
Note, if screen is not available, you can install it (on RHEL/Fedora systems) via:
yum install screen
If you're running into SSH timeouts in general, there's another thing you can do - set your client's ServerAliveInterval. While you can do other things (I'm often infinitely lazy and simply run a command with I/O, such as top), this is an elegant solution. Simply add the following to your ~/.ssh/config file:
This will make your SSH client send a keepalive packet to the remote server every thirty minutes, and generally should keep your connection from prematurely closing. Note that you may have to play around with that value, depending on the configuration of the server.