PS1 IPP Czar Logs for the week 2011.11.07 - 2011.11.13

(Up to PS1 IPP Czar Logs)

Monday : 2011.11.07

Mark is czar

• 09:00 ipp036 removed from nebulous so Cindy can powerdown and replace CPU+heatsink
• 09:30 remaining night data having 503 error in summitcopy (c5872g0011o--c5872g0014o). finished @10:00
• 11:30 MD07-y nightly stack sample running in normal stack pantasks to verify skycell coverage with the new center shift for exposures since 4/2009.
• 13:15 ipp036 back up, putting back into nebulous fully and slowly adding back into processing
• 13:30 ipp053 slowly putting back into processing, taken out Friday for over weekend due to mount issues and many tasks wanting to run on it.
• 13:45 ipp036 export not up, ran exportfs -f
• 14:00 ippc13 slowly add back processing since watching ipp036,053 too. was taken out from processing because of second crash w/in hour over weekend.
• 14:45 ippc10 also back online, adding back in for processing
• 15:30 running MD07.reftest.20111107 sample of different number of input warps in skycell.055 for all filters with the deepstack pantasks (compute2 group).
• 16:30 cleaning LAP -- needed to do more often, now have 1500 stacks to run..

  -- troubled update fault=2, find update log neb://ipp020.0/gpc1/LAP.ThreePi.20110809/2011/10/09/o5463g0339o.229905/o5463g0339o.229905.ch.317278.XY31.log.update and problem with with neb://ipp044.0/gpc1/20100924/o5463g0339o/o5463g0339o.ota31.fits
        0                     NON-EXISTANT file:///data/ippb02.1/nebulous/10/c3/967714255.gpc1:20100924:o5463g0339o:o5463g0339o.ota31.fits
        0                     NON-EXISTANT file:///data/ippb02.1/nebulous/10/c3/967715079.gpc1:20100924:o5463g0339o:o5463g0339o.ota31.fits
  
   chiptool -updateprocessedimfile -set_state full -chip_id 317278 -class_id XY31 -dbname gpc1
   chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 317278 -class_id XY31 -dbname gpc1
  
  -- more update trouble chip_id=339071, drop from multiple runs - 
  laptool -updateexp -lap_id 1705 -exp_id 229905 -set_data_state drop -dbname gpc1
  laptool -updateexp -lap_id 1706 -exp_id 229905 -set_data_state drop -dbname gpc1
  laptool -updateexp -lap_id 1709 -exp_id 229905 -set_data_state drop -dbname gpc1
  laptool -updateexp -lap_id 1710 -exp_id 229905 -set_data_state drop -dbname gpc1
  
  -- warp (update?) trouble for exp_id=260145, drop from runs
  laptool -updateexp -lap_id 1699 -exp_id 260145 -set_data_state drop -dbname gpc1
  laptool -updateexp -lap_id 1700 -exp_id 260145 -set_data_state drop -dbname gpc1
  laptool -updateexp -lap_id 1703 -exp_id 260145 -set_data_state drop -dbname gpc1
  
  -- warp fault due to chip file problem
  Reading FITS file /data/ipp033.0/nebulous/a6/76/1544401387.gpc1:LAP.ThreePi.20110809:2011:11:07:o5523g0098o.260145:o5523g0098o.260145.ch.339377.XY52.ch.fits failed.
  
  perl ~ipp/src/ipp-20110622/tools/runchipimfile.pl --chip_id 339377 --class_id XY52 --redirect-output
  

• 16:40 info note: running lap_science.pl --monitor_mode --lap_id 1730 reported

  QUALITY: 223457 has bad warp quality: 16 / 77
  STATUS: LAP_ID           1730
  STATUS: DEFINED_QSTACK:  0
  STATUS: HAVE_QSTACK:     0
  STATUS: DEFINED_FSTACK:  0
  STATUS: HAVE_FSTACK:     0
  STATUS: NEEDS_REMADE:    0
  STATUS: NEEDS PRIVATIZE: 0
  STATUS: ARE WARPED:      102
  STATUS: ARE MAGICKED:    50
  STATUS: CAN_DIFF:        101
  STATUS: HAVE_DIFF:       54
  STATUS: TOTAL_EXPOSURES: 113
  
  means LAP kicking out that exposure (currently done if >20% warps have bad quality). 
  

• 17:30 ippc13 okay so far with the 4x stdscience, 2x stack so adding back distribution,pstamp,update,publish
• 18:00 ippc13 wend down, so seems can handle the stdscience+stack load but not distribution. will try some combos to see what is stable. Chris also recommended maybe adding a compute_broken group in ~ipp/ippconfig/pantasks_hosts.input that could also include ippc11 (seems to handle 3 processes okay).
• 19:00 ippc13 looks like crashed when the 1x update started while running 2x stdscience, 2x stack. drop down to 2x stdscience, 1x stack and group with ippc11 in hosts_poor_compute
• 20:30 another LAP chip lost

  Nebulous key neb://ipp043.0/gpc1/20100619/o5366g0539o/o5366g0539o.ota25.fits is consistent.
        0                     NON-EXISTANT file:///data/ippb02.0/nebulous/4c/7e/893019408.gpc1:20100619:o5366g0539o:o5366g0539o.ota25.fits
        0                     NON-EXISTANT file:///data/ippb02.2/nebulous/4c/7e/893026715.gpc1:20100619:o5366g0539o:o5366g0539o.ota25.fits
  

• 21:00 ippc11 donw. cannot run 1x stack + 2x stdscience so remove from processing completely again. rebooting, but not coming back up. leaving off.
• 23:30 ippc11 powered up.
• 23:40 LAP clearing all of sunday's (11/6) runs now and morving forward.
• MD07.refstack.20111106 z-band etc running under the deepstack pantasks. MD07.GR0 sample nightly stacks added to the main stack pantasks as they run quicker and at a lower priority that can be interrupted by LAP.

Tuesday : 2011.11.08

• 11:50 Mark: stare nodes been quiet for a while, adding back into processing. distribution uses 4 nodes on each of the stare nodes, maybe 1-2 of these each could be added to stack since distribution seems to be keeping up.
• 15:00 ipp038 raid back to allocate in nebulous. Bill pointed out ipp038 was in nebulous state repair, Mark had put it into repair 11/5 when raid reported degraded with disk issue. finished rebuilding 11/6 just forgot to re-enable.

Wednesday : 2011.11.09

• down most of the afternoon to test trunk for potential new tag
• 16:42 Bill restarted pantasks
• 16:47 fixed magic fault due to corrupt diff skyfile --diff_id 188156 --skycell_id skycell.0879.049
• 16:48 fixed destreak fault due to corrupt chip file --chip_id 300733 --class_id XY65 Wait a second. That file must have been used for a warp. Why wasn't it corrupt then???

Thursday : 2011-11-10

Bill is czar today

• 09:20 fixed a corrupt warp file warp_id 303159 --skycell_id skycell.2243.044
• 09:25 dropped 3 chip LAP chip files that are repeatedly getting a SEGV: -chip_id 340549 -class_id XY67 -chip_id 340551 -class_id XY67 -chip_id 340552 -class_id XY67
• 10:20 Hayden replaced ippdb02's RAID battery backup unit.
• He also took ippc11 and ipp029 down for examination to see if there is some obvious reason that they have been crashing frequently.
• 10:30 started using the new tag ipp-20111110
• 11:00 ipp029 back on line. Started up the other pantasks
• 11:07 Fixed the first bug in the tag (mine in magic_destreak.pl)
• 13:37 ipdpb02 is back: BBU has been replaced and is fully charged (according to Gavin). Write cache is enabled
• 13:37 Postage stamp server is broken. There is a memory management problem in Ohana in the new tag. Worked around it by changing dvoImagesAtCoords to leak a tiny bit of memory.
• 16:30 Mark: runs in deepstack pantasks finished. shutdown and restarted with new tag.
• 16:35 new MD03 from last night with the new MD03.V3 refstacks looks good except for the WS diffims, will wait to see the SS diffims (keep forgetting to fix the time when those are run..)

Friday : 2011-11-11

Saturday : 2011-11-12

Sunday : 2011-11-13