PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.04.18

• MEH:
  • moving stdscience pantasks back to its original ippc02 node so not sharing ippc01 s/ summitcopy
  • testing survey.pro changes for making the SSdiff have a long needed target label arg..
  • testing summitcopy time stamps in log
  • turning off stack nodes to compare processing rate during next MD07 night
  • running ~ipp pantasks log bzip

Tuesday : 2016.04.19

• 15:30 MEH: boosting pstamp to get through a large backlog of QUB stamps before nightly processing

Wednesday : 2016-04-20

• 14:45 CZW: I'm disabling the wave 3 nodes that are still in use in the pantasks_hosts.input file, and will be adding 3x x2 to stdscience, and 1x x3 to all the other pantasks that had 1x s2.
• 15:30 CZW: ipp/pantasks restarted with new pantasks_hosts.input file. All servers now have equal or slightly higher number of active nodes than before.

Thursday : YYYY.MM.DD

• 13:30 CZW: Stopped pantasks for config SVN repair.
• 13:30 CZW: Rebooting wave 3 machines.
• 13:50 CZW: ipp033 rebooted but does not see the home directory. I am not rebooting any more machines at this time.
• 15:00 CZW: Rebooting has resumed. Beginning safety rsyncs on ipp033, ipp035, and ipp036.
• 23:50 MEH: pstamp seems to be hosed due to some change today -- ~ipp/ippconfig/site.config was modified to an incompatible version

  #default access level for users that do not have specific entries in the tables
  PSTAMP_DEFAULT_ACCESS_LEVEL S32 1
  --> required!
  
  PSTAMP_PRESERVE_DAYS STR 14 -> 30
  
  --> should be commented out while ipp017 is down? not sure what problems that would cause
            CATDIR.017     STR    /data/ipp017.0/ipp/ippRefs/catdir.refcat.20140713.v0
  

Friday : 2016.04.22

• 01:55 MEH: confirmed rate drop during longer exposure CFA/MD07 observations and not from stack processing, will turn stack back on in morning to run normally
• 17:02 HAF: restart of pantasks - notes that there are jammed jobs (dsget / regimfile) on ipp104 that jammed at exactly the same time as a bunch of rsyncs started..

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD