[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.07.16 ===
Bill is czar today.
 * 10:10 stdscience was stuck. There are several thousand chip runs outstanding but no jobs are getting queued. Czartool indicates that chip processing stopped making progress around 20:15 last night while outstanding warps got finished just after midnight. The book chipPendingImfile had lots of entries in init state but nothing was getting run. I restarted stdscience.
 * 13:40 Chris discovered that ippc03 was out of space in /tmp which was causing nebulous requests to fail. This undoubtedly contributed to the processing failures. Bill deleted the log files and restarted the apache servers. 
 * 13:55 chip reverts are off because we have a number of exposures that are faulted repeatedly due to incorrect burntool state. Chris is working on fixing the burntool states.
  
=== Tuesday : 2012.07.17 ===

=== Wednesday : 2012.07.18 ===

=== Thursday : 2012.07.19 ===

=== Friday : 2012.07.20 ===

=== Saturday : 2012.07.21 ===

=== Sunday : 2012.07.22 ===