PS1 IPP Czar Logs for the week 2011-02-28 - 2011.03.06

(Up to PS1 IPP Czar Logs)

Monday : 2011.02.28

Serge is czar.

• Bill stopped stdscience last night and feel asleep before restarting it. (We had a burst of faults that he wanted to investigate. Revert took care of them.) Restarted stdscience about 7:30 am.
• 14:25 Restarted Replication. It was stopped Sunday night around 20:37. Note that stdscience crashed around 8:30pm as well. Its log many errors (these errors becoming more frequent from 20:05 till the crash) like:

  500 Can't connect to ippdb00.ifa.hawaii.edu:80 (connect: timeout) at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/lib/Nebulous/Client.pm line 852
  

  I checked the different syslogs (apache, mysql, /var/log/messages) but I didn't see anything interesting in them.

Tuesday : 2011.03.01

Bill is czar.

• about 4am queued another set of ThreePi.136 diffs. As advertised the script takes a long time
• 7:30 running low on outstanding diffs. Ran the script again.
• 7:50 many faults. Log shows ippc20 can't access files from ipp037. force.umount fixed that problem. Reverted faults.
• 10:30 EAM : removed ippc00 - ippc04 from stdscience to use for static sky photometry
• 11:21 heather set chip/warp/stack of MD07.trp and MD06.jtrp to goto_cleaned, and added MD08.jtrp to ipp/stdscience.
• 11:21 heather started the diff queue script for ThreePi.136
• 12:30 EAM : ippc02 hung on over-full memory. i've rebooted it.
• 16:23 heather ran the diff queue script again.

Wednesday : 2011.03.02

Bill is czar.

Thursday : 2011.03.03

Gene is czar.

Friday : 2011.03.04

Gene is czar.

Saturday : 2011.03.05

Sunday : 2011.03.06