Changes between Version 26 and Version 27 of PS1_IPP_Czarlog_20131216

Timestamp:

Dec 21, 2013, 12:24:47 AM (13 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20131216

@@ -73 +73 @@
  mark is czar
  * 09:00 MEH: pausing everything to trace down what was stalling processing late last night -- this includes all extra processing.. 
-  * [wiki:CZW_notes] has details of some of the extra processing
+  * [wiki:CZW_notes] has details of some of the extra processing -- turning off 
 {{{
+--- convolved stack cleanup -- uses compute3
+pantasks_client -c ~watersc1/this_is_where_pantasks_lives/ptolemy.rc
+
+--- rawcheck running in replication using wave2-4 + 2_weak
+pantasks_client -c ~ipp/replication/ptolemy.rc
+}}}
+  * ippc01,c02,c03 -- /var/log/apache2/error_log having a growing number of "server seems busy" messages recently -- only c01-c03, are these being manually extra targeted?
+{{{
+[Fri Dec 20 11:05:25 2013] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 21 total children
+}}}
+   * restarted apache on all 3, cleared larger nebulous_server.log on ippc02
+{{{
+sudo /etc/init.d/apache2 stop
+sudo rm /tmp/nebulous_server.log ; sudo touch /tmp/nebulous_server.log ; sudo chown apache /tmp/nebulous_server.log ; sudo chmod g+w /tmp/nebulous_server.log ; ls -l /tmp/*log
+sudo /etc/init.d/apache2 start
+}}}
+  * ipp044 was running neb_rawOTA_host_scan.pl with long >10ks query times in the nebulous DB -- kill -STOP to pause, but appears to have died or finished (on last set?)
+{{{
+| 19865251 | ipp      | ipp044.ifa.hawaii.edu:48902  | nebulous | Query       |   15915 | Sending data                                                   | SELECT ins_id,so_id,uri FROM instance WHERE vol_id = 39 AND ins_id > 4280476494 LIMIT 10000 | 
+
+| 19865251 | ipp      | ipp044.ifa.hawaii.edu:48902  | nebulous | Query       |   11300 | Sending data                                                   | SELECT ins_id,so_id,uri FROM instance WHERE vol_id = 39 AND ins_id > 4284170025 LIMIT 10000
+
+--- cmdline when kill -STOP
+watersc1  2130 30746  0 Dec16 pts/0    00:01:06 perl ./neb_rawOTA_host_scan.pl --limit 10000 --continue --host ipp044 --min 2479121495
 
 }}}
+ * 13:10 MEH: things moving more now but still reduced..
+ * 14:00 nope, breaking down again..  
+ * 15:20 MEH: last night found ~1/3 LAP runs (20770-20775) were stalled from messed up chip state goto_cleaned when warps full -- all chips still untouched, set back to full and LAP cleared for stacks
+{{{
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925815
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925808
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925809
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925810
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925811
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925812
 
-  * 
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925813
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 924307
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925814
 
- * 13:10 MEH: things moving more now but still reduced..
- * 14:00 nope, breaking down again.. ipp044 doing heavy neb queries?
- * 15:20 MEH: last night found ~1/3 LAP runs were stalled from messed up chip state goto_cleaned when warps full -- 
+chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925813
+}}}
  * 22:00 MEH: no nightly so far, preparing for regular stdscience restart once LAP triggers stacks
+  * unclear source of problem last night, LAP rate now is >100/hr and seems higher than have had recently -- possibly bad intersection of too many file operation processes, '''may want to turn some of these extras off during nightly'''
+  * manually turned off nodes again -- ipp031,032 (rsyncs still?); 015, 050 (generally problematic, need reboot soon?); ipp042, 043, 044, 046, 047, 048 (running neb_rawOTA_host_scan.pl)
+
 === Saturday : 2013-12-21 ===