Changes between Version 43 and Version 44 of PS1_IPP_CzarLog_20110103

Timestamp:

Jan 7, 2011, 8:34:37 AM (15 years ago)

Author:

Serge CHASTEL

Comment:

--

PS1_IPP_CzarLog_20110103

@@ -71 +71 @@
 
 === Friday : 2011.01.07 ===
+serge is czar
+
  * (bills) 05:00 1 file repeatedly failing summit copy and 1 repeatedly failed registration. The problem is that nebulous instances for the image file (copy) and registration log file have been created on ipp025 but that node has been taken out of nebulous. ganglia says that ipp025 has a high load. I can log into it. On ippdb00 I tried force.umount and it successfully unmounted the ipp025 but the remount step never finished. To unstick registration and summit copy I used neb-mv to move the inaccessible instances out of the way.
- * 05:15 summit copy is finished but registration/burntool has 134 unfinished exposures. It seems to be slowly making progress burntooling files. But no files are finishing registration. The burntool jobs are for chips from exposures after o5568g0183o (the one that faulted) so perhaps something is wrong. One chip (XY35) from o5568g0183o has burntool_state == -1 That is probably blocking things from proceeding. I don't see a regtool mode to fix this so I edited the database and set burntool_state back to zero. That didn't seem to help though.
- * 05:55 We needed data_state changed from check_burntool to pending_burntool. Now we're moving along. Sounds like we may need a revertburntool mode. 
- * 06:05 Stopped registration to make rawImfile.burntool_state a key.
- * 06:30 power cycled ipp025 There was a panic message on console. See https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ipp025_log
- * 06:39 reverted warp, diff, and dist faults (probably caused by ipp025) 
- * 06:40 added ThreePi.nightlyscience back into stdscience label list. (Gene removed it last night)
- * 07:00 registration/burntool has finished
- * 07:11 Someone submitted a postage stamp by coordinate request through the web interface for a point in M31. CFA has some requests that are blocked by that so I've lowered the priority for the postage stamp label WEB and gpc1 label ps_ud_WEB to let the cfa requests through.
+ * (bills) 05:15 summit copy is finished but registration/burntool has 134 unfinished exposures. It seems to be slowly making progress burntooling files. But no files are finishing registration. The burntool jobs are for chips from exposures after o5568g0183o (the one that faulted) so perhaps something is wrong. One chip (XY35) from o5568g0183o has burntool_state == -1 That is probably blocking things from proceeding. I don't see a regtool mode to fix this so I edited the database and set burntool_state back to zero. That didn't seem to help though.
+ * (bills) 05:55 We needed data_state changed from check_burntool to pending_burntool. Now we're moving along. Sounds like we may need a revertburntool mode. 
+ * (bills) 06:05 Stopped registration to make rawImfile.burntool_state a key.
+ * (bills) 06:30 power cycled ipp025 There was a panic message on console. See https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ipp025_log
+ * (bills) 06:39 reverted warp, diff, and dist faults (probably caused by ipp025) 
+ * (bills) 06:40 added ThreePi.nightlyscience back into stdscience label list. (Gene removed it last night)
+ * (bills) 07:00 registration/burntool has finished
+ * (bills) 07:11 Someone submitted a postage stamp by coordinate request through the web interface for a point in M31. CFA has some requests that are blocked by that so I've lowered the priority for the postage stamp label WEB and gpc1 label ps_ud_WEB to let the cfa requests through.
+ * (serge) 08:34 chip.off to speed up MD
 
 === Saturday : 2011.01.08 ===