Changes between Version 37 and Version 38 of PS1_IPP_CzarLog_20110103

Timestamp:

Jan 7, 2011, 5:51:54 AM (15 years ago)

Author:

bills

Comment:

--

PS1_IPP_CzarLog_20110103

@@ -71 +71 @@
 === Friday : 2011.01.07 ===
  * (bills) 05:00 1 file repeatedly failing summit copy and 1 repeatedly failed registration. The problem is that nebulous instances for the image file (copy) and registration log file have been created on ipp025 but that node has been taken out of nebulous. ganglia says that ipp025 has a high load. I can log into it. On ippdb00 I tried force.umount and it successfully unmounted the ipp025 but the remount step never finished. To unstick registration and summit copy I used neb-mv to move the inaccessible instances out of the way.
- * 05:15 summit copy is finished but registration/burntool has 134 unfinished exposures. It seems to be slowly making progress. However it is working on chips from exposures after o5568g0182o (the one that faulted) so perhaps something is wrong.
+ * 05:15 summit copy is finished but registration/burntool has 134 unfinished exposures. It seems to be slowly making progress burntooling files. But no files are finishing registration. The burntool jobs are for chips from exposures after o5568g0183o (the one that faulted) so perhaps something is wrong. One chip (XY35) from o5568g0183o has burntool_state == -1 That is probably blocking things from proceeding. I don't see a regtool mode to fix this so I edited the database and set burntool_state back to zero. That didn't seem to help though.
 
 === Saturday : 2011.01.08 ===