== PS1 IPP Czar Logs for the week 2011-02-28 - 2011.03.06 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.02.28 ===
Serge is czar.
 * Bill stopped stdscience last night and feel asleep before restarting it. (We had a burst of faults that he wanted to investigate. Revert took care of them.) Restarted stdscience about 7:30 am.
 * 14:25 Restarted Replication. It was stopped Sunday night around 20:37. Note that stdscience crashed around 8:30pm as well. Its log many errors (these errors becoming more frequent from 20:05 till the crash) like:
{{{
500 Can't connect to ippdb00.ifa.hawaii.edu:80 (connect: timeout) at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/lib/Nebulous/Client.pm line 852
}}}
   I checked the different syslogs (apache, mysql, /var/log/messages) but I didn't see anything interesting in them.
=== Tuesday : 2011.03.01 ===
Bill is czar.
 * about 4am queued another set of ThreePi.136 diffs. As advertised the script takes a long time
 * 7:30 running low on outstanding diffs. Ran the script again.
 * 7:50 many faults. Log shows ippc20 can't access files from ipp037. force.umount fixed that problem. Reverted faults.
 * 10:30 EAM : removed ippc00 - ippc04 from stdscience to use for static sky photometry
 * 11:21 heather set chip/warp/stack of MD07.trp and MD06.jtrp to goto_cleaned, and added MD08.jtrp to ipp/stdscience.
 * 11:21 heather started the diff queue script for ThreePi.136
 * 12:30 EAM : ippc02 hung on over-full memory.  i've rebooted it.

=== Wednesday : 2011.03.02 ===
Bill is czar.

=== Thursday : 2011.03.03 ===
Gene is czar.

=== Friday : 2011.03.04 ===
Gene is czar.

=== Saturday : 2011.03.05 ===

=== Sunday : 2011.03.06 ===