id summary reporter owner description type status priority milestone component version severity resolution keywords cc 1009 pcontrol crashes when node crashes Paul Price eugene "ipp003 went down (presumably the usual ethernet problem), which triggered a pcontrol crash (see below). Node crashes have been correlated with pcontrol crashes in my mind for some time, but wanted to get this down for reference and tracking. Note that when pcontrol goes down, it appears to come back up, but the state (in particular, the connections to the pclients) is not preserved, so that processing does not continue. pantasks: controller status job stack PENDING: 14 objects 0 anyhost 11 warp_skycell.pl --warp_id 21 --skycell_id skycell.0199092 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2ea [...] job stack BUSY: 6 objects 0 anyhost 11 warp_skycell.pl --warp_id 21 --skycell_id skycell.0199087 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2e9 1 anyhost 11 warp_skycell.pl --warp_id 21 --skycell_id skycell.0199085 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2e7 2 anyhost 11 warp_skycell.pl --warp_id 20 --skycell_id skycell.0198673 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2c9 3 anyhost 11 warp_skycell.pl --warp_id 21 --skycell_id skycell.0199086 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2e8 4 anyhost 11 warp_skycell.pl --warp_id 21 --skycell_id skycell.0199083 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2e5 5 anyhost 11 warp_skycell.pl --warp_id 20 --skycell_id skycell.0198671 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.2c7 job stack DONE: 0 objects job stack KILL: 0 objects job stack EXIT: 0 objects job stack CRASH: 0 objects host stack OFF: 0 objects host stack DOWN: 0 objects host stack IDLE: 0 objects host stack BUSY: 6 objects 0 bucket00 0.0.0.4 1 ipp002 0.0.0.0 2 ipp003 0.0.0.1 3 bucket00 0.0.0.5 4 bucket00 0.0.0.3 5 ipp003 0.0.0.2 host stack DONE: 0 objects pantasks: got signal : 13 controller is down starting controller connection Connected pantasks: status Scheduler is running Controller is running [...] Jobs in Pantasks Queue 23: warp.skycell.run 720 warp_skycell.pl (790990) 23: warp.skycell.run 740 warp_skycell.pl (7ed900) 23: warp.skycell.run 28 warp_skycell.pl (80bed0) [...] jobs currently running remotely: Njobs: 0 pantasks: controller status job stack PENDING: 1 objects 0 anyhost 11 warp_skycell.pl --warp_id 22 --skycell_id skycell.0199092 --tess_id MOPS --camera MEGACAM --workdir path://MOPS/run1/ 0.0.0.0 job stack BUSY: 0 objects job stack DONE: 0 objects job stack KILL: 0 objects job stack EXIT: 0 objects job stack CRASH: 0 objects host stack OFF: 0 objects host stack DOWN: 0 objects host stack IDLE: 0 objects host stack BUSY: 0 objects host stack DONE: 0 objects" defect assigned high PanTasks unspecified major