OpenMPI v.1.2.4 built using Intel's compilers v.11 (using CaOS' cports, very nice).

n0008 spiraled down with a large number of accumulating non-completing ssh commands (originating from house-keeping temperature and load monitoring cron scripts). Failed to identify the source of the problem, failed to pass commands in any other way, hard reboot was the (non-)solution. Pooh.