Changes between Version 21 and Version 22 of PS1_IPP_Czarlog_20190510

Timestamp:

May 14, 2019, 2:24:26 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -474 +474 @@
  I have continued to feed updates to the update beast. However, it was not very clear to me what the current and upcoming order of things is. therefore, I tried to write up a (quick) query for checking the number of exposures that 'need doing' within %LAP.PV3% labels. To be clear, this means the number of exposures with a camRun.label like %LAP.PV3% for which the most recent (re-)processing is in a %cleaned or update state. The query is:
 {{{
-mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,camRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'
+mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'
 }}}
  and the resulting list is:
@@ -725 +725 @@
 }}}
  
+ ---------------------
+
+ Czar meeting at 13:30:
+ We want to test the ippx nodes, now that they have been redistributed within their cabinet. Best way, set them as the only nodes to be used by stdscience for the ~2hours until the daily restart. Look at hosts groups in ippconfig/pantasks_hosts.input to find which host groups are loaded by the regular stdscience pantasks:
+{{{
+macro hosts.stdscience
+  echo hosts.stdscience
+  control run reap
+
+  hosts.ignore.storage
+
+## 5x, 4x seems to work better, fewer faults  MEH 20170316
+## reduce 3x si0, boost 5x ci0+1 while dvo/psps running MEH 20170812
+## adding more power with data nodes s4-8 after large MOPS update testing MEH 180524
+   for i 0 3   
+     hosts add si0
+     hosts add s4
+     hosts add s5
+     hosts add s6
+     hosts add s7
+     hosts add s8
+   end
+  # 
+  for i 0 5 
+    hosts add ci0
+    hosts add ci1
+  end
+  # add another ci1 but not ci0 since ippitc+pantasks running there too MEH 180524
+  hosts add ci1
+
+  control run
+end
+}}}
+ Now then, to get the x nodes in, we first added the hosts definitions for the ippx nodes to the ippconfig/pantasks_hosts.input file. Then, we restarted the pantasks to load those in, and then use 'hosts off' with the appropriate groups to turn off the current set of nodes. Then, we add hosts groups x0,x1,x2 a total of 10 times, to get their CPUs fully utilised. They are using about 80 percent CPU on ganglia but seem to be behaving alright.
+ Keep an eye on boottime in ganglia, to see if any x nodes randomly reboot, in prep for PS2. Tomorrow morning, add the xnodes back in to test them again, and keep an eye on them.