PS1 IPP Czar Logs for the week 2015.01.19 - 2015.01.25

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.19

• 05:25 EAM: ipp062 crashed around 04:15, nothing on console. processing completely stalled. i'm rebooting it now.
• 05:40 EAM: after the reboot of ipp062, processing back up and running.
• 09:55 MEH: ipp084 is in WT state and may be dragging on processing, into repair
• 10:20 MEH: two warps stalling OSS finish

  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369542 -skycell_id skycell.1609.051
  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369710 -skycell_id skycell.1609.051
  

• 14:45 EAM: stopping & restarting stdlocal
• 15:15 EAM: rebooting ippx016 -- load very high for no good reason, ganglia shows there has been no I/O since midnight -- I suspect an ethernet crash.
• 15:20 EAM: oops: I just shutdown stdlocal accidentally. restarting it again.

Tuesday : 2015.01.20

• 05:00 EAM: stdscience is running sluggishly; stopping it to restart it
• 10:20 EAM: some warps with growth curve failures:

  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371670 -skycell_id skycell.1697.023
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371761 -skycell_id skycell.1698.077
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371765 -skycell_id skycell.1700.047
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371802 -skycell_id skycell.1787.048
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371947 -skycell_id skycell.1701.070
  

• 12:00 EAM: ipp041 crashed, nothing on console. rebooting.
• 12:20 EAM: ipp056 has hung mounts, trying to clear

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD