Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20180827

Timestamp:

Aug 29, 2018, 9:36:06 AM (8 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20180827

@@ -6 +6 @@
 === Monday : 2018.09.02 ===
 
- * 10:15 EAM : ippb04 and ippb05 are up allowing us to have a b-node replication target.  I've set them up for now and am re-running the MOPS dailytestset chunk, which now seems to be succeeding.
+ * 10:15 EAM : ippb04 and ippb05 are up allowing us to have a b-node replication target.  I've set them up for now and am re-running the MOPS dailytestset ~~chunk~~ quad, which now seems to be succeeding.
  * MEH: summitcopy & registration getting allocation of 1x s4 (ipp067-70) to substitute for down compute nodes and test data node use there
- * MEH: updating stdscience, pstamp, ippqub:stdscience_ws in master ~ippitc/ippconfig/pantasks_hosts.input now 
- * MEH: ippb06-b15 to repair to allow chips to be updated for stamps -- due to there being replicated products having copies only on the backup nodes...
+ * MEH: updating stdscience, pstamp, ippqub:stdscience_ws in master ~ippitc/ippconfig/pantasks_hosts.input now from their input files 
+ * MEH: ippb06-b15 to repair to allow chips to be updated for stamps -- due to there being replicated products having copies only on the backup nodes still...
  * MEH: more test reprocessing for MOPS without masks with label/data_group mehtest.nomask
  * MEH: moving pstamp pantasks from ippc67 to ippc33 -- testing use of lower ippc nodes and network with pantasks (try to trigger network issue seen with lower ippc and ipp123-126)
-=== Tuesday : YYYY.MM.DD ===
 
-=== Wednesday : YYYY.MM.DD ===
+=== Tuesday : 2018.08.29 ===
+ * MEH: more broken/missing LAP/2014 files flushed out with pstamp test -- also noticing some nfs/cpu wait issues for ipp124 to new ipp127-ipp129 nodes in normal stdscience updates, may be due to over requests of older data on those machines
+  * no apparent ssh interruptions/faults however (extra loadings of ipp123-126 in pstamp)
+ * MEH: Weryk starting nice'd processing again -- ipps nodes to begin with
+
+=== Wednesday : 2018.08.30 ===
+ * MEH: ipp113 root disk nearly out of space and will bork all kinds of things... -- both apache and messages very very large
+  * manually do log rotation after stopping apache
+  * no nagios warning -- need to verify if that is setup for this node -- we need ping/alive, root+export disk space check on all nodes basically, maybe also load?
+ * MEH: ippx001-x020 root disk also nearly or actually out of space... -- do manual log rotation there like ippx021-x088 a few weeks ago
+ * MEH: ippb07-15 neb-host repair->up now since seem stable after shutdown
 
 === Thursday : YYYY.MM.DD ===