IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 78 and Version 79 of PS1_IPP_Czarlog_20120917


Ignore:
Timestamp:
Sep 22, 2012, 2:27:26 PM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20120917

    v78 v79  
    6262 * 14:10 after all these reboots, Serge manually restarting the mysql for dneb-locate.py to use again (ipp010,011,020,037,061)
    6363 * 15:00 MEH: fixing stalled LAP runs to clear some of the older ones (goto_cleaned put some in an odd state) ~150 chips
    64  * 18:10 MEH: will stop processing in ~30 min to shuffle load a bit again. stare and compute/2 nodes get hammered with stacks at times. Manual plan after restart
    65   * stare: -2 stk +2 stdsci
    66   * c: -1 stk
    67   * c2: -1 stk (instead of stdsci), still overloading so -2 now
    68   * w4: -1 stk (maybe +1 stdsci)
    69   * c3(deep): +1 stdsci (as before, but if stk becomes underpowered, switch to there but stks can clear during nightly science time)
    70   * w1: -1 stdsci
    71   * -3x ipp028 had problems in past -- ipp020 probably fixed so put back in the 3x from before, not sure about ipp028 so keep out
    72  * 19:50 MEH: ipp044 has nfs problem -- rebooting.. all processing stalled
    73   * took a while to reboot, raid spin-up?
     64 * 18:10 MEH: will stop processing in ~30 min to shuffle load a bit again. stare and compute/2 nodes get hammered with stacks at times.
     65  * memory usage summary
     66  * Manual plan after restart
     67   * stare: -2 stk +2 stdsci
     68   * c: -1 stk
     69   * c2: -1 stk (instead of stdsci), still overloading so -2 now
     70   * w4: -1 stk (maybe +1 stdsci)
     71   * c3(deep): +1 stdsci (as before, but if stk becomes underpowered, switch to there but stks can clear during nightly science time)
     72   * w1: -1 stdsci
     73   * -3x ipp028 had problems in past  -- ipp020 probably fixed so put back in the 3x from before, not sure about ipp028 so keep out
     74 * 19:50 MEH: ipp044 has nfs problem -- rebooting.. all processing stalled  [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Ipp044-nfs-20120921]
     75  * took a while to reboot (3-5 min), raid spin-up?
    7476  * restarted registration and something is borked.. going again after ran {{{ regtool -revertprocessedimfile -dbname gpc1 }}}
    7577 * 20:30 compute2 overloaded, by stack still. -1 added above list for total of 2.