== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.05.07 ===
 * 07:25 Mark: registration stalled? had to run to get things moving along
{{{
regtool -updateprocessedimfile -exp_id 484476 -class_id XY43 -set_state pending_burntool -dbname gpc1

-- seems to also be burntool taking 20ks but not going to try and fix until get in
  0    ipp008    BUSY  21216.52 0.0.3.740b  0 ipp_apply_burntool_single.pl --exp_id 484466 --class_id XY43 --this_uri neb://ipp015.0/gpc1/20120507/o6054g0187o/o6054g0187o.ota43.fits --continue 10 --previous_uri neb://ipp015.0/gpc1/20120507/o6054g0186o/o6054g0186o.ota43.fits --dbname gpc1 --verbose 
-- nothing in registration log for this. looks like stalled on ipp008 in a neb-replicate for neb://ipp015.0/gpc1/20120507/o6054g0197o/o6054g0197o.ota43.burn.tbl
-- which has 3 entries now and one with different md5sum -- culled 0 size one and didnt unstick
-- neb-replicate not able to kill. not going to mess with anymore, emailed czar, maybe just needs to restart replication pantasks. otherwise mostly everything registered and processing
}}}

 * 10:00 heather: messing with addstars...
 * 14:05 Bill rebooting ipp008. It isn't playing nicely with others. Also will restart stdscience.
=== Tuesday : 2012.05.08 ===
Bill is czar today.
 * 07:30 restarted summit copy. Set ipp008 to repair. It is bogging things down.
 * 09:56 summit copy is proceeding but running into "503 try again" faults from conductor
 * set newExp for exp_id 456051 456052 from December 15 to state 'drop' They keep reverting and failing to register. c5972g0035l and c5972g0036l
=== Wednesday : 2012.05.09 ===
 Mark czar today
 * 07:30 looks like nightly science finished downloading and processing.
 * 11:15 Chris rebuilt ops tag and restarted stdscience to reprocess SAS with changes (details)
=== Thursday : 2012.05.10 ===
 Mark is czar
 * 07:40 all nightly data downloaded and processed except for one chip on ipp008 running for 36ks. manually killed ppImage on ipp008, reverted and continued through warp fine. not a particularly crowded field at all, nothing stands out of the normal in the ipp008 log around the time it was originally running.
 * 10:30 connections from manoa to production cluster seem to have been interrupted/lost for a bit (as was found earlier in the morning when got in). ippc19 seems to not be coming back or is down. 
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===