| | 8 | We have been dealing with a number of hardware problems this week, now with machines that have not had problems before. We had several of the ippcXX nodes crash (these have in the past been quite reliable). ippc22 now seems to be unusable for processing -- it crashes if any load is put on it. In addition, ipp029 has been having trouble since the previous week -- we (Cindy) tried replacing memory modules with no luck. It also cannot be used for any processing or as a nebulous target -- I now suspect a flaky raid card. Up at the ATRC, one of the raids on ippb00 was behaving badly; we have a replacement raid card on order. Finally, ipp036 crashed hard on Thursday and could not be rebooted. With the training going on on Oahu and Cindy needed at the summit on Friday, we could not get to this until Monday. Cindy replaced the motherboard -- it seems a cpu fan died and took out the mobo as well. There is some suspicion that we have having trouble with heat management at MHPCC, but we do not have enough data monitoring to know. We will update ganglia with per-machine temperatures. |
