PS1_IPP_Czarlog_20150518 – Pan-STARRS IPP

Context Navigation

Version 9 (modified by eugene, 11 years ago) ( diff )
--

16:23 CZW: restarting ippsky/pv3ff{lt|rt} because they appear to have been running forever.

05:45 EAM: stdscience is running somewhat slowly. i'm restarting it.
12:10 MEH: an abundance of ps_ud_QUB label diffs in goto_cleaned state not being cleaned again -- changing label to goto_cleaned to free space
20:00 EAM: restarting fforce
- 20:45 EAM: we are getting some excessively large fforce jobs on pv3ffrt from the Galactic plane. I'm removing storage nodes from that pantasks until we are past.

05:30 EAM: rebooting stsci03 -- processing is somewhat behind, but not yet clear if that is due to stsci03. (the reboot helped -- things are now moving along)
06:10 EAM: I'm restarting the pv3fflt,pv3ffrt pantasks. pv3ffrt is working on the plane -- it is slower than pv3fflt and it needs more memory per job. at the same time, pv3fflt is constantly running out of jobs because the queuing is barely able to keep up. I'm adjusting the loading to put all but 14 of the x nodes in pv3ffrt (since they have more memory), and leaving both storage and compute to pv3fflt (when stdscience does not need them) along with the 14 unused x nodes.
11:00 EAM: I allowed too many jobs to the x2 and x3a nodes for pv3ffrt : they are thrashing on too much memory. I am trying to kill off jobs and bring them down to 5 jobs instead of 9.