IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20130610

Version 17 (modified by Mark Huber, 13 years ago) ( diff )

--

PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

    1. PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD
      1. Monday : 2013.06.10
      2. Tuesday : 2013.06.11
      3. Wednesday : 2013.06.12
      4. Thursday : 2013.06.13
      5. Friday : 2013.06.14
      6. Saturday : 2013.06.15
      7. Sunday : 2013.06.16

(Up to PS1 IPP Czar Logs)

Monday : 2013.06.10

  • reminder @0830 power off to IfA-Manoa

Tuesday : 2013.06.11

  • RCUH holiday
  • 13:15 MEH: Serge has pointed out pstamp running slowly lately for MOPS requests, looks like pantasks running since 5/29. doing restart of pstamp and update, will see if rate back up tomorrow morning/next MOPS round
  • 18:55 MEH: also did restart of summitcopy, registration since running for while as well. stdscience probably needs to be done no later than tomorrow.

Wednesday : 2013.06.12

Bill is czar today

  • 05:50 Quite a bit of data was taken last night. 18 object exposures left to download. 95 exposures still at chip stage. Ganglia looks nominal except that network usage has a not quite uniform sawtooth pattern with period of about 5 minutes.
  • 06:12 Chip stage throughput is only 30 exposures per hour. stdscience pcontrol is spinning. Going to restart stdscience. Restarted at 06:22

Thursday : 2013.06.13

Bill is czar today

  • 05:20 disk space is getting low. Set ps_ud% and STS.rp.20130508 data to goto_cleaned
  • 14:00 MEH: starting the select MD09 reprocessing for MD09.refstack.2013, manually adding deepstack compute3 nodes back into stdsci
  • 19:30 MEH: stdsci chip/warp @60k jobs run and Nrun underloaded.. seems sooner than typical for regular restart.. doing before nightly starts..
  • 19:45 MEH: ippc18/home disk 36G free..
  • 21:00 MEH: never fails.. md09 reprocessing almost done and system goes belly-up... ipp057 having trouble, load >150.. neb-host down, trying to take out of processing..
    • 21:20 had to kill 5xppImage, 1x pspwarp, reducing stdsci load to it -- unclear what happened, like other wave4, mysql ~50% or more of RAM and looks like fell into 100% system CPU for something
  • 21:50 MEH: reconfiguring deepstack back to doing deep stacks only, taking 4-of-6x compute3 out of stdscience again

Friday : 2013.06.14

  • 01:00 MEH: md09 reftests faulting often in stack creation with I/O error code: 32cd.. appear to be ippc40,c44, maybe c53 often having the faults. unclear why, didn't happen with md08 in April. turning on revert overnight, maybe most will get done eventually...

Saturday : 2013.06.15

Sunday : 2013.06.16

Note: See TracWiki for help on using the wiki.