== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : YYYY.MM.DD === === Tuesday : 2017.08.08 === * MEH: another LAP stack missing files and blocking WSdiffs, set qual 42 to clear * MEH: QUB stamps stalled since 8/7 due to fault 3, manually cleared === Wednesday : YYYY.MM.DD === === Thursday : YYYY.MM.DD === === Friday : 2017.08.11 === * MEH: large number of summit faults from Dell switch fallover (see Sidik email) -- fault 202 to clear manually === Saturday : 2017.08.12 === * MEH: chip stage processing has been having some very large times (camera to a lesser extent) and appear to be slowing the nightly processing rate -- manually adjusted ippitc/stdscience node allocation, removing si0 2x and adding ci0+1 2x roughly seemed to help last night -- monitoring * si0 nodes have dvo/psps processing and seems to be multiple rsync jobs for dvo to ippb nodes from Gene tonight (unsure how long that has been running, ipp117 is critical at night) * refcat also only on ipp105,107-110 -- adding back in ipp054-059,061-063,065-066 to site.config, rsync'ing it back === Sunday : YYYY.MM.DD ===