PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.11.17

• 01:20 MEH: ippsXX in use for MD PV3 -- ~ippmd/stdscience/ptolemy.rc if need to turn off
• 12:50 MEH: going to start using ippx001-020 for MD processing rate tests, turning off slowly in stdlocal
  • ippx013 many faults -- Gavin fixed missing entries in autofs

Tuesday : 2014.11.18

• 07:30 MEH: stdsci underloading, will need a restart after nightly finishes along w/ summit and reg. removing update labels for now -- was well underloaded so restarted
  • lanl stdlocal also seems to be using storage nodes when nightly not finished, 2x stacks on upp039 (only 8 cores as well..) was causing chips to run ~5x longer than normal.. manually turning storage nodes off..
• 09:45 MEH: nightly finally finished
• 09:30 MEH: ippsXX and ippxNNN off for ippmd, turned on in lanl stdlocal for SAS processing
• 10:55 EAM: restarted stdlocal with the following modified logic for loading:
  • storage hosts are either all on (daytime, 6AM HST - 7PM HST) or all off (nighttime)
  • chip-warp processing is blocked only if > 10k nightly science items are outstanding
  • ippxNNN nodes are loaded in 2 blocks: x0 and x1, corresponding to machines on the same PDU
  • ippxNNN and ippsNN nodes are not ON by default, but are loaded for ease of pcontrol operations
  • added the SAS.20141118 label to process the newest SAS run.
• 11:10 MEH: turning ippsXX and ippxNNN on then in stdlocal for SAS prioity
• 21:35 MEH: registration backed up >100 because of a fault?

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD