PS1 IPP Czar Logs for the week 2014.10.13 - 2014.10.20

(Up to PS1 IPP Czar Logs)

Monday : 2014.10.13

• if pixelservers shutdown described in PS1 shutdown list, will want to be sure all nightly data downloaded by early morning
• 06:00 MEH: nightly data downloaded
• 09:00 MEH: no nightly data until ~Wednesday -- disable nightly shutoff of nodes in lanl stdlocal
  • looks like lanl stdlocal needs a regular restart anyways
  • after restart will see if this works tonight or if there is something else that also needs to be flipped.. -- server input tweak_offstorage
• 12:03 Bill: restarted pstamp pantasks and queued ps_ud label for cleanup.
  • also dropped warp_id 453790 whose corresponding camRun 472138 was in state drop, which prevents updates for that warpRun from succeeding.
• 13:41 Bill: turned MPE label off in pstamp. There is a large backlog of requests to be cleaned for that label and the new ones are running so fast that we are falling behind on space.
  • 14:15 cleanup is done 75% full. Adding MPE label back in.

Tuesday : 2014.10.14

• 09:20 MEH: Ken has noted lanl stdlocal has been pretty much idle since ~4am -- looks like handful mem fault warps and many fault 2 stacks can try to be cleared. actually only one was mem fault, three are this and don't know what has been done for this case

  cannot build growth curve (psf model is invalid everywhere)
  Backtrace depth: 11
  Backtrace 0: p_psAssert
  Backtrace 1: pmGrowthCurveGenerate
  Backtrace 2: psphotMakeGrowthCurve
  Backtrace 3: psphotChoosePSFReadout
  Backtrace 4: psphotChoosePSF
  Backtrace 5: psphotReadoutFindPSF
  Backtrace 6: (unknown)
  Backtrace 7: (unknown)
  Backtrace 8: (unknown)
  Backtrace 9: __libc_start_main
  Backtrace 10: (unknown)
  Assertion failed in function pmGrowthCurveGenerate at pmGrowthCurveGenerate.c:89. Error stack:
  

• 10:45 CZW: restarting stdlocal, stopping stdlanl to do LANL side cleanup.

Wednesday : YYYY.MM.DD

• 8:45 CZW: All ipp/ipplanl pantasks servers shutdown in preparation for MRTC-B power outage.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD