Changes between Initial Version and Version 1 of Production_Cluster_Status

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

Production_Cluster_Status

@@ +1 @@
+<pre>
+
+Status as of 2008-11-14
+
+Nodes down:
+
+None.
+
+Notes:
+
+- ipp015 is rebuilding disk #16
+
+Known issues:
+
+- random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler
+  (device driver) for it.
+- ipp004 - disk bay #12 dead (system is usable)
+- ipp005 - seems to have random instability issues beyond the do_IRQ issue of the other nodes
+- ipp008 - disk bay #7 dead (system is usable)
+- ipp013 & ipp016 have dead fans but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
+- forcedeth.c max_interrupt_work is still too conservative (requires a cluster restart to change).
+- ipp025 - suspect bad memory module (system is usable)
+- Our contact at MHPCC, Brad Thomas, notified me that an alarm from cabinet 4 managed power strip had been set off last evening due to high loads on nodes (ipp017, 030-036).
+  Please consider distributing your jobs evenly across ipp production nodes.
+
+  Below is the layout at MHPCC.
+  cabinet0: ipp020-21
+  cabinet1: ipp012-014,008,016,018-019
+  cabinet2: ipp004-007,015,009-011
+  cabinet3: ipp023-029
+  cabinet4: ipp030-036,017
+
+http://kiawe.ifa.hawaii.edu/IPPwiki/index.php/Production_Cluster_Status
+
+</pre>