PS1_IPP_CzarLog_20110131 – Pan-STARRS IPP

Context Navigation

Version 13 (modified by bills, 16 years ago) ( diff )
--

Bill has magic cleanup is running cleaning up old magic runs on ipp053. This is the reason for the relatively higher load. pantasks directory is ~bills/magic_cleanup.

Bill 06:55 Lots of red on the board this morning. This could be because there is a lot of update processing going on. 3 stack faults. Turned stack.revert on (since we fixed the known bugs it should be on all the time now.)
Bill 07:00 Once last night's data is through warp I will queue the next STS night from September. We still have several thousand STS exposures to deliver. This key project has been much neglected.
Lots of database problems (probably due to multiple processing streams) and nfs errors today.
Bill 17:24. Two distRuns fail to complete because of corrupted input files. dist_id: 409008 403036 Set state to 'hold' to stop the fail/revert/fail cycle
Set STS.201009 to be distributed.
17:30 set update and distribution pantasks to off to see if the load drops significantly. Yep. Set them to run at 17:40
fixed a corrupted warp file with runwarpskyfile. I lost the ids. the diff that was failing completed.
15:50 STS is through diff. So the only thing running is distribution and update). Still getting lots of nfs faults.
restarting distribution pantasks to reduce the number of enabled hosts.
we got an MDO07 exposure last night and we are threatened with MD06. Added these labels back into stdscience survey tasks
17:00 Bill is investigating distribution and destreak failures. dist.revert.off and destreak.revert.off
all reverts back on
fixed one corrupt warp component with perl ~bills/ipp/tools/runwarpskycell.pl --redirect-output --warp_id 157463 --skycell_id skycell.298

05:50 (Bill) space page on czartool is mostly red. STS.20100909 is done except for 3 broken diff stage distRuns. Dropped those and set chip,warp,and diff data to be cleaned.

Note: See TracWiki for help on using the wiki.