PS1 IPP Czar Logs for the week 2011.06.13 - 2011.06.19

(Up to PS1 IPP Czar Logs)

Monday : 2011-06-13

• 10:20 Restarted replication
• 10:55 : Chris queued-up a single MD04 exposure for re-processing as requested by Jim
• 11:10 : CZW Begin shuffle speed up test by doubling number of hosts running jobs.
• 12:00 : Bill queued chip runs for a new STS reference stack: label STS.refstack.20110613
• 12:09 : Bill queued STS.2009.b data that is finished to be cleaned. Queued all outstanding ps_ud data to be cleaned.
• 13:24 : Bill noticed that warp was stuck (all entries in warpPendingSkyCell had pantaskState DONE) warp.reset
• 13:30 : Bill rebuilt a broken diff skycell that was causing a magic error rundiffskycell.pl --redirect-output --diff_id 138137 --skycell_id skycell.2080.074
• 13:31 : Bill fixed a broken streakMap file by setting the offending magicNodeResult to fault = 2. Upon revert it will be regenerated.
• 13:35 : Bill 25 tries is enough difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 138027 -skycell_id skycell.1635.016

Tuesday : 2011-06-14

• 06:10 Restarted replication
• 12:34 : Bill warp 208424:skycell.2.20:STS.V3 was in the warpPendingSkyCell book yet there was no job in the controller list. Deleted the page.

Wednesday : 2011-06-15

• 04:48: sts refstack exposures are through warp. Queued the stacks. 438 of 448 skycells got queued. The missing ones are on the edges of the projection centers.
• 08:25: sts.2009.b data queued so far is on distribution server. Set the data to be cleaned.
• 15:45: The value of NEB_SERVER has been changed in ~ipp/.tcshrc and ~ipp/ippconfig/site.config. If necessary revert to the previous value (ippc03)
• 17:12: heather - bill found some faults for me to investigate. They all clustered around the same time (near 4pm), and looked nebulous related, so I reverted it and it worked.

Thursday : 2011-06-16

• 10:45: MPG has all of the STS data on the data store. Set the distRuns' to be cleaned.
• 12:54: a magic node was faulting repeatedly due to a corrupted diffskyfile fixed with perl ~bills/ipp/tools/rundiffskycell.pl --diff_id 138666 --skycell_id skycell.2259.100 --redirect-output
  • 13:00: Set label STS.nightlyscience to allow STS.refstack diffs to complete. We want to insure that the masking fraction doesn't increase going from warp-warp diffs to warp-stack diffs

Friday : 2011-06-17

Saturday : 2011-06-18

Sunday : 2011-06-19