| | 369 | '''OTA03 issues''' by JRF |
| | 370 | |
| | 371 | OTA03 decided it had had enough and went crazy, causing some major stalling during the night in the chip stage (jobs stalling in pantasks for well over 2000 seconds). |
| | 372 | |
| | 373 | To fix this the jobs need to be killed and moved to quality 42 (since this is manual change). |
| | 374 | |
| | 375 | Fortunately, there is handy script that MEH has that does this. Check out: |
| | 376 | {{{ |
| | 377 | ~mhuber/bin/sshkill_ppimage_dtime_host_ps1.csh |
| | 378 | }}} |
| | 379 | The gist of it is that is looks at the long running chip jobs in pantasks, then ssh's to each machine for the jobs and runs another kill script to stop ppimage.bat. After this, it waits 20 seconds for them to clear and goes and sets the jobs -- which are all in state 'new' and fault 4 -- to fault 0 and quality 42. |
| | 380 | |
| | 381 | I've temporarily added this to the crontab on ippc30 (since this is where a bunch of other ippitc crontabs are). I'm waiting to see if it works |
| | 382 | |
| | 383 | |
| | 384 | |
