PS1 IPP Czar Logs for the week 2015.09.07 - 2015.09.13

(Up to PS1 IPP Czar Logs)

Monday : 2015.09.07

• 07:45 MEH: phantom reboot of ipp008,12,13,14,16,18 again last night ~0210 HST -- they seem to have come back up okay
• 10:05 MEH: restarting nightly pantasks
• 10:20 MEH: oddly seeing >400 chips w/ label goto_clleaned for data_group x.20150728 -- don't see in my cut/paste script notes so not sure who from.. -- then seeing few MD.PV3 here and there.. looks like a pstamp bug?
• 20:25 MEH: looks like obs stalled from weather, clearing stalled WS diffs -- 37 skycells lost from 18 exposures
• 21:00 MEH: MOPS finished w/ x.20150905, all-clear for data to be cleaned up

Tuesday : 2015.09.08

• 01:04 MEH: clearing bad warp -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.1129.012 -warp_id 1619498  
  

• 01:11 MEH: cleared WS diffs again -- 601 skycells lost from 78 exp
• 06:00 MEH: ipp017 crashed ~2hrs ago, nothing on console and not unusual load -- power cycling
  • keyboard error -- had to skip w/ F1
  • many ypbind failure -- finally connected and jobs clearing from stdsci as well @0615 back to normal processing w/ ipp017 out and back to repair
• 07:55 MEH: clearing ws diffs -- 664 skycells lost for 208 exp
• 16:40 MEH: Haydn rebooted ipp017 to try and stop the PCI warning -- seems to be fine again, neb-host repair again but still manually out of processing

  [RAID-SYSTEM ALARM]
  2015-9-9  6:30:58 Controller#1(PCI) SATA PHY +2.5V   Over Voltage
  

• 22:00 MEH: clearing bad warp -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.1221.028 -warp_id 1619871
  

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD