Hung Linux system: Now what?
Finally, I had remembered to try out Magic SysRq key in real life. One of the XenServer hosts decided to refuse to reboot and hung during shutdown. Magic SysRq is really a keyboard combo that allows user to force kernel to perform certain function, even though machine is unresponsive. In my case I needed to safely reboot the host.
For Magic SysRq to work, it must be enabled in kernel:
[root@db0004 ~]# cat /proc/sys/kernel/sysrq 1
If output is 1, then you are good to go. Thankfully, XenServer had this enabled. So, while holding down Alt and SysRq keys I pressed the following keys: r e i s u b.
So, what do all the letters mean? I ripped the explanation from Wikipedia. Article is worth checking out, especially the section on how to deal with keyboards other than QWERTY.
r : Switch the keyboard from raw mode, the mode used by programs such as X11 and svgalib, to XLATE mode
e : Send the SIGTERM signal to all processes except init (PID 1)
i : Send the SIGKILL signal to all processes except init
s : Sync all mounted filesystems
u : Remount all mounted filesystems in read-only mode
b : Immediately reboot the system, without unmounting or syncing filesystems
The Wikipedia article also mentions couple of mnemonics to help remembering the key sequence.