== PS1 IPP Czar Logs for the week 2014.09.29 - 2014.10.05 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.09.29 ===
* 07:27 Bill: Set quality flag on several diffs that were failing
{{{
difftool -updatediffskyfile -fault 0 -set_quality 14006 -diff_id 598546 -skycell_id skycell.2163.039
difftool -updatediffskyfile -fault 0 -set_quality 14006 -diff_id 598547 -skycell_id skycell.2162.028
difftool -updatediffskyfile -fault 0 -set_quality 14006 -diff_id 598547 -skycell_id skycell.2162.057
difftool -updatediffskyfile -fault 0 -set_quality 14006 -diff_id 598607 -skycell_id skycell.1155.020
difftool -updatediffskyfile -fault 0 -set_quality 14006 -diff_id 598617 -skycell_id skycell.1246.074
}}}

=== Tuesday : 2014.09.30 ===
 * 12:00 MEH: using ippsXX for MOPS test processing -- done

=== Wednesday : 2014.10.01 ===

=== Thursday : 2014.10.02 ===
 * 10:25 EAM : ipp013 is back up after a mobo-replacement.  I've reverted the failed chips, but I've also run neb-repair on the ipp013-only burntables that were needed.
 * 11:20 CZW : restarted stdlanl pantasks, as it had a lot of timeout/failure counts that made reading the number of active jobs tricky.

=== Friday : 2014.10.03 ===

 * 06:10 EAM : ipp036 crashed in the night, no messages on the console. I power cycled and it came back fine.
 * 10:55 EAM : burntool was having trouble with a single chip (o6933g0296o.ota14).  It turns out there is a problem with an array in burntool which is not forced to be consistent with an accessing loop variable.  I made a hackish fix, but have asked JT for help
 * 23:10 MEH: noticing the fault 2 build up again -- lanl stdlocal running ~230 jobs alongside nightly science
{{{
 -> psFitsOpen (psFits.c:217): I/O error
     Failed to delete a previously-existing file (/data/ipp082.0/nebulous/54/bb/5186700861.gpc1:OSS.nt:2014:10:04:o6934g0265o.804031:o6934g0265o.804031.ch.1117466.XY27.mdl.fits), error 2: No such file or directory
 -> pmFPAfileOpen (pmFPAfileIO.c:816): I/O error
     error opening file /data/ipp082.0/nebulous/54/bb/5186700861.gpc1:OSS.nt:2014:10:04:o6934g0265o.804031:o6934g0265o.804031.ch.1117466.XY27.mdl.fits
 -> pmFPAfileWrite (pmFPAfileIO.c:415): I/O error
}}}
 * 23:20 MEH: some fault 5 diffims to clear so MOPS gets them in the morning
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2616.002 -diff_id 599551  -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2617.028 -diff_id 599551  -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2616.060 -diff_id 599556  -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2617.065 -diff_id 599556  -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2617.068 -diff_id 599556  -fault 0
}}}
=== Saturday : 2014.10.04 ===
 * 00:17 MEH: no problems w/ remote connection now after Gavin's email about the firewall fix. few more diffim fault 5 to clear
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1234.088 -diff_id 599564 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1235.079 -diff_id 599565 -fault 0
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1324.032 -diff_id 599571 -fault 0
}}}
=== Sunday : 2014.10.05  ===