Changes between Version 22 and Version 23 of PS1_IPP_Czarlog_20190426

Timestamp:

May 25, 2019, 11:23:52 AM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -16 +16 @@
 === Sunday : 2019.04.28 ===
  * MEH: since pstamp removed from daily restarts because auto-restart script still does not respect parsing jobs and this corrupting them and pantasks still needs to be restarted when Njobs~100k, have added a crontab on ippitc@ippc26 (where pstamp pantasks runs) that checks every hour if nearling ~100k and needs to be restarted (also with note if parsing jobs running)
-
-
-'''OTA03 issues''' by JRF
-
-OTA03 decided it had had enough and went crazy, causing some major stalling during the night in the chip stage (jobs stalling in pantasks for well over 2000 seconds).
-
-To fix this the jobs need to be killed and moved to quality 42 (since this is manual change).
-
-Fortunately, there is handy script that MEH has that does this. Check out:
-{{{
-    ~mhuber/bin/sshkill_ppimage_dtime_host_ps1.csh
-}}}
-The gist of it is that is looks at the long running chip jobs in pantasks, then ssh's to each machine for the jobs and runs another kill script to stop ppimage.bat. After this, it waits 20 seconds for them to clear and goes and sets the jobs -- which are all in state 'new' and fault 4 -- to fault 0 and quality 42.
-
-I've temporarily added this to the crontab on ippc30 (since this is where a bunch of other ippitc crontabs are). I'm waiting to see if it works
-
-