[hendi_]

Yes, it's more complicated than just SSH'ing into the server.

But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you.

Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.

[rdl]

Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.

[hendi_]

Could you please elaborate on that? How do you do "trusted" live forensics on systems with possibly infected kernels and stuff? Assuming these servers were normal COTS and nothing fancy (thinking of CPU-bypassing memory access...)

[jevinskie]

DMA memory out using FireWire (if available)? That would be my approach!

[rdl]

Firewire is awesome for the attacker, unfortunately few servers have it, especially not externally exposed ports. Also, smart OSes use some of the newer Intel features (VT-d) to lock down DMA while the OS is running, which usually protects from rogue firewire, and can theoretically help against rogue PCIe, although usually badly implemented in chipset and OS.

Another option is a reboot onto a custom OS which is designed specifically to preserve memory (you get a safe few seconds of holdover). LiveKd is pretty cool (sysinternals)

There are PCIe cards which do processor/network and let you explore main memory -- WindowsSCOPE CaptureGUARD for PCIe or ExpressCard. Probably enough time to pop the case open and throw one in before memory degrades.

Countermeasures are numerous -- everything from doing memory encryption inside the CPU die (putting code in the cache, like TRESOR) and doing hypervisor tricks ("TresorVisor") (http://www1.informatik.uni-erlangen.de/tresor) to using Hardware Security Modules (like the SafeNet or Thales nCipher) to just keeping your servers physically secured from intruders who might memory-analyze them (although a software bootloader and remote-reboot could still be applied).

Forensics as a field seems to be a lot more interested in attacking mobile phones (which is one of the things I'm talking about at RSA 2013), but desktops and servers are still interesting targets.