PS1 IPP Czar Logs for the week 2014-02-24 - 2014-03-02

(Up to PS1 IPP Czar Logs)

Monday : 2014-02-24

• 10:37 Bill set stdscience to stop. It is desperately in need of a restart.
  • serge came by and reported that the postage stamp server is slow. Restarted it and added a set of compute3 nodes.
  • 10:44 stdscience restarted with standard (low power host set)
  • restarting staticsky
• 14:45 CZW: quick note on how I investigated the issue from last night:
  • Step 1: check that registration was working correctly: regpeek.pl
  • Step 2: investigate the state of diffs: nightly_science.pl --debug --verbose --queue_diffs --date 2014-02-24 This command does not actually execute any commands, but does the full logic check to see what's up. The interesting line is: diff_queue: Number of input warps to make diffs is not even for target OSS and object ps1_14_1783! 4 : I should declare an exposure to be qualityy. , which despite the spelling error, points out that object ps1_14_1783 has 5 warps, and so it needs to exclude one (because diffs require an even number of exposures). However, this is done based on exp_id (for historical reasons), which kicks out the visit 1 instance.

Tuesday : 2014-02-25

• 10:54 Bill: daily restart of pstamp server
• 11:27 Bill: stopping staticsky in preparation to rebuild the tag to support distribution of full force runs
  • back to run I will do this after lunch.
  • 12:59 set to stop
• 16:30 Bill: rebuilt ippsky's ipp and started up a pantasks to distribute the full force runs. This shouldn't take too long to complete.
  • 16:38 set staticsky to run
• 17:30 CZW: In an attempt to keep stdscience from falling behind, I've made the following changes:
  • stdscince:

     hosts add wave2; hosts off ignore_wave2
     hosts add wave2; hosts off ignore_wave2
     hosts add wave3; hosts off ignore_wave3
     hosts add wave3; hosts off ignore_wave3
     hosts add compute2; hosts off ignore_compute2
    

  • staticsky:

     hosts off wave3
    

Wednesday : 2014-02-26

• 05:10 Bill: processing is pretty far behind again. Set staticsky to stop. Rate may have climbed a little bit.
  • 06:10 summit copy backlog 67
• 07:30 Bill: added 2 sets of compute3 nodes to stdscience removed LAP label stdscience
  • pcontrol is spinning I'm not sure that pantasks is keeping things loaded as well as it could. status commands mostly timnig out.
  • really really need to do a daily restart
• 07:53 now we are getting a very large number of database related faults. detselect commands are failing. ipptool -pending queries are timing out.
• 07:58 Bill: I'm going to restart stdscience
  • 08:05 restarted stdscicence adding the deepstack hosts (staticsky is off)
• 10:13 Bill: with permission from the czar (gene) I am adding the STS label to stdscience. I will watch for memory problems. (I could not reproduce the result that we got on Friday)
  • the memory problem is back. It seems that some chips (XY24 and XY31 at least) for some exposures gets really whacky results, generating lots of detections (>300,000), which causes memory to grow very large (>20GB) when writing the cmf file. Unfortunately the targets for these chips (ipp016 and ipp020) have 24 GB of memory so they can't handle multiple ppImage instances. Further debugging is required
• 10:55 Gene is going to restart things without the STS label.
• 10:55 EAM : I found a situation which may be causing pcontrol to get extra slow and sticky : hosts which appear to be hung. I've added a bit of debugging to understand this better, so I'm stopping everything and restarting. In restarting, I'm keeping the STS label out since it was was causing some trouble with memory explosions (2009 data on a specific chip).
• 16:33 Bill: ippdb01 has apparently gone down. Setting pantasks to stop

Thursday : 2014-02-27

Friday : 2014-02-28

Saturday : 2014-03-01

Sunday : 2014-03-02