== IPP MHPCC Production Cluster Status ==

(Up to [wiki:IPP_for_PS1 IPP for PS1])

'''Status as of 2008-11-14'''

=== Nodes down ===

None.

=== Notes === 

 * ipp008 raid is being completely rebuilt
 * ipp014 is now using the chassis from ipp017 (which had been ipp037, but was sent back to ASA for repair)
 * ipp037 is using the chassis from ipp014
 * ipp037 
 * ipp015 is rebuilding disk !#16

=== Known issues === 

 * ipp005 - occasional crashes -- we suspect the memory may be bad.  '''note: as of 2009.10.30, the memory has been replaced; we need to stress this machine and see if it continues to crash'''
 * ipp008 - complete raid failure 2009.10.28 -- disks have been replaced and raid needs to be re-built.  ipp008 has not been used for storage for some time, so we have not lost any vital data.
 * ipp009 - several recent crashes with coincident complaints from CPU !#3.
 * ipp013 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
 * ipp014 - has a newer version of the Wave !#1 motherboard which requires older kernel drivers 
 * ipp016 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
 * ipp037 - fails to boot; suspect motherboard
 * ipp025 - suspect bad memory module (system is usable)

=== Old Issues ===

 * random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler
  (device driver) for it
  * these were only present on Wave !#1 hardware
  * these seem to have largely stopped with the current forcedeth driver and kernel

 * ipp004 - disk bay !#12 was dead, but started working again after the disk was reseated (ipp004 has device labels inverted, and it is suspected that previous re-seat attempts were addressing the wrong disk)

 * cabinet 4 managed power strip was being overloaded.  With the wave 3 delivery, we rebalanced the power load to avoid this problem.

===  Layout at MHPCC ===
{{{
  cabinet0: ipp020-21
  cabinet1: ipp012-014,008,016,018-019
  cabinet2: ipp004-007,015,009-011
  cabinet3: ipp023-029
  cabinet4: ipp030-036,017
}}}