Changes between Version 19 and Version 20 of PS1_IPP_Czarlog_20150105

Timestamp:

Jan 8, 2015, 10:52:17 AM (12 years ago)

Author:

watersc1

Comment:

--

PS1_IPP_Czarlog_20150105

@@ -38 +38 @@
  * 07:45 EAM: it looks like cab1 had a hiccup last night.  all machines in that cab, ex ipp009 and ipp019, rebooted about 9pm.  I stopped all processing to clear out some mount point troubles.  then it turned out that ipp018 was having xfs problems on /export/ipp018.0.  I tried to repair, but needed to reboot to umount.  on boot, /dev/sda4 refused to mount.  I ran xfs_check /dev/sda4 and it said the metadata needed to be replayed by mounting.  mounting gave an error that "Structure needs cleaning".  I ran xfs_repair /dev/sda4 as advised, and it said that the log was unrecoverable.  I am now running xfs_repair -L /dev/sda4 as advised, and it is trying to clean up the disk partition.  The output is going to /root/sda4.repair.log.  I have since restarted the processing with ipp018 in neb 'down'.  there will probably be some errors until ipp018 is up if anything put output there (was it in neb up earlier? not sure).  
  * 09:10 EAM: xfs_repair finally finished and /export/ipp018.0 mounted successfully.  nfs exports it just fine now.  
-
+ * 10:43 CZW: processing is working, but there are a lot of faults caused by failures to read detrends on ipp018.  The files exist, have the correct md5sum, and funpack correctly.  A closer look suggests that we have mounts on some of the x-nodes that can't correctly read from ipp018.  ippx090 is an example of this:  attempting to read a file results in 'no such file' errors, and the mount point doesn't mount correctly.  I'm going to set ipp018 to down to try and get nightly science to clear out, and when ipp018 is quiet, see if a reboot makes it behave again.
+ 
 === Friday : 2015.01.09 ===