== PS1 IPP Czar Logs for the week 2013.02.18 - 2013.02.24 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.02.18 ===
 * 08:00 MEH: regular restart of stdscience to keep LAP rate up
 * 22:00 MEH: running low on chip/warp/diff (pile-up of stacks in GP) so adding a few more runs 43->50
=== Tuesday : 2013.02.19 ===
 mark is czar
 * 08:00 MEH: again restarting stdscience to keep the LAP rate up
 * 08:40 MEH: ipp008-ipp010 taken out of processing in ppconfig/pantasks_hosts.input for Gene to use with ipptopsps runs 
 * 08:50 MEH: chip.revert.off while fixing LAP chips
 * 09:30 MEH: giving stack an extra allocation of compute3 to try and more quickly push the pile of 3000 through -- without nightly science to process stacks never caught up over the weekend
 * 10:50 MEH: out of chips/warps.. adding more LAP runs -- should've used the ~ipp/lap/trickle_add.pl script to slowly add 1x1, but was manually doing similarly so should be okay. in the past Chris notes some exposures get dupe processed.
 * 10:57 Bill: ran 'queuessky.lap --ra_min 60 --ra_max 90 --go' which queued about 5800 five filter staticsky runs
  * MEH: set poll 30->10 until some of the large backlog of stacks move through since staticsky running in stack now
 * 16:30 MEH: ippc03 has had warnings about disk space, there is/was an old old 2011   nebulous_server.old.log hanging onto 24G of disk
 * 16:40 turning skycal.on back and in the stdscience/input file
 * 17:50 Chris setup next RA hour slice for LAP
 * 18:30 MEH: ippdb01 went into heavy WAIT_CPU state and all processing piled-up. stopping processing until worked out 
 * 19:40 MEH: things settling down, slowly restarting
 * 23:25 MEH: LAP stacks mostly caught up so reallocating all back to default
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===