PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.24

• 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
• 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
• 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
• 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
• 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products.
• 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though.

Tuesday : 2011-10-25

• 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing perl runchipimfile.pl --chip_id 331016 --class_id XY63
• 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
• 09:45 warp.revert.off (need some time to fix the flaky cam files). perl runcameraexp.pl --cam_id 307180
• 10:00 warp.revert.on
• 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems
• 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..). the files involved look okay, instance and filesizes are correct.

  gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits
  gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits
  gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits
  gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits
  gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits
  gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits
  gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits
  gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits
  gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits
  gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits
  gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits
  gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits
  gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits
  gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits
  gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits
  gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits
  gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits
  

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD