PS1_IPP_Czarlog_20180827 – Pan-STARRS IPP

Context Navigation

Version 9 (modified by Mark Huber, 8 years ago) ( diff )
--

10:15 EAM : ippb04 and ippb05 are up allowing us to have a b-node replication target. I've set them up for now and am re-running the MOPS dailytestset ~~chunk~~ quad, which now seems to be succeeding.
MEH: summitcopy & registration getting allocation of 1x s4 (ipp067-70) to substitute for down compute nodes and test data node use there
MEH: updating stdscience, pstamp, ippqub:stdscience_ws in master ~ippitc/ippconfig/pantasks_hosts.input now from their input files
MEH: ippb06-b15 to repair to allow chips to be updated for stamps -- due to there being replicated products having copies only on the backup nodes still...
MEH: more test reprocessing for MOPS without masks with label/data_group mehtest.nomask
MEH: moving pstamp pantasks from ippc67 to ippc33 -- testing use of lower ippc nodes and network with pantasks (try to trigger network issue seen with lower ippc and ipp123-126)

MEH: more broken/missing LAP/2014 files flushed out with pstamp test -- also noticing some nfs/cpu wait issues for ipp124 to new ipp127-ipp129 nodes in normal stdscience updates, may be due to over requests of older data on those machines
- no apparent ssh interruptions/faults however (extra loadings of ipp123-126 in pstamp)
MEH: Weryk starting nice'd processing again -- ipps nodes to begin with then move on to ippx and ippc30-63 as before

MEH: ipp113 root disk nearly out of space and will bork all kinds of things... -- both apache and messages very very large
- manually do log rotation after stopping apache
- no nagios warning -- need to verify if that is setup for this node -- we need ping/alive, root+export disk space check on all nodes basically, maybe also load?
MEH: ippx001-x020 root disk also nearly or actually out of space... -- do manual log rotation there like ippx021-x088 a few weeks ago
MEH: ippb07-15 neb-host repair->up now since seem stable after shutdown