PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017.08.08

• MEH: another LAP stack missing files and blocking WSdiffs, set qual 42 to clear
• MEH: QUB stamps stalled since 8/7 due to fault 3, manually cleared

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : 2017.08.11

• MEH: large number of summit faults from Dell switch fallover (see Sidik email) -- fault 202 to clear manually

Saturday : 2017.08.12

• MEH: chip stage processing has been having some very large times (camera to a lesser extent) and appear to be slowing the nightly processing rate -- manually adjusted ippitc/stdscience node allocation, removing si0 2x and adding ci0+1 2x roughly seemed to help last night -- monitoring
  • si0 nodes have dvo/psps processing and seems to be multiple rsync jobs for dvo to ippb nodes from Gene tonight (unsure how long that has been running, ipp117 is critical at night)
  • refcat also only on ipp105,107-110 -- adding back in ipp054-059,061-063,065-066 to site.config, rsync'ing it back

Sunday : YYYY.MM.DD