Changes between Version 4 and Version 5 of ITC_shuffle

Timestamp:

Dec 1, 2016, 12:09:53 PM (10 years ago)

Author:

watersc1

Comment:

--

ITC_shuffle

@@ -33 +33 @@
 The initial testing (currently at 8119 of 10000 jobs) has an average execution time of 426.89s, but with a very long tail.  I manually killed ~10 glockfile processes that prompted the ippb05 exclusion, so those jobs have execution times of ~60000s.  The current 'controller status' output shows jobs still running at 600-700s, with no clear reason for the slowdown.  Some of the initial jobs were run previously during testing, but that should only comprise ~1500 jobs at most.
 
+=== Problem cases ===
+
+The main source of problems that I've noticed is due to mounting issues.  Usually only one node has issues, and even then, it's usually only with a single mount point.  Running {{{umount -f /data/ipp0XY.Z}}} sometimes clears this up, after killing all rawcheck.pl instances running on that problem node (sometimes killing glockfile instances are also necessary).  However, when this doesn't work, restarting nfs on that problem node (the one with the hung mount, not the NFS server) seems to sort things out: {{{/etc/init.d/nfs restart && /etc/init.d/rpc.statd restart}}}.
+
+
+
 === Possible mitigation in case of speed concerns ===