(Up to PS1 IPP Czar Logs)

Monday : 2012.09.24

• CZW: 12:00 Stopped system for stdscience reboot.
• CZW: 12:04 Rebooted ipp020 due to mounting issues that were resistant to force.umount. Rebooted using the call:

  sync
  sync
  sync
  reboot -f
  

• CZW: 12:10 autofs was not correct on ipp020. /etc/init.d/autofs restart resolved the problem.
• CZW: 12:12 Repeated this procedure on ipp063, once ipp020 returned.
• CZW: 12:16 Restarted stdscience.
• CZW: 13:50 Restarted ipp020/gmond to fix the "ganglia purple memory bug" that Mark pointed out.

Tuesday : 2012.09.26

• 08:00 Bill We got some data last night. registration has a stuck job on ipp010. It is having the "ganglia purple memory bug" Killed the register_imfile. Rebooting ipp010
• 08:05 reset registrations pending book. set fault for the ipp010 victim 521411:ota26 then reverted it. burntool is now feverishly working to catch up.

Wednesday : 2012-09-26

Serge is czar

• 09:00 (Serge): rsync mysql@ippdb02 to ippc63 complete. Starting rsync of mysql@ippc63 to ippc61
• 15:10 MEH: turning off the deepstack compute3 group from stack pantasks for a local/trunk deepstack pantask setup for MD09.GR0 staticsky run
• 16:25 CZW: restarting stdscience because it looks slow.
• 23:40 MEH: looks like ipp018 has lost some mounts, removing from processing to try and get nightly science back online and avoid rebooting.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD