Changes between Version 25 and Version 26 of PS1_IPP_Czarlog_20190204

Timestamp:

Feb 11, 2019, 2:03:27 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190204

@@ -320 +320 @@
 }}}
 
-JRF: ME, MEH, and TdB shall be going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area).
-
-
-
 * 16:00 EAM : I started a set of rsyncs on ippb04 to copy nebulous directories to ippb16, ippb18, ippb19 (for b04.0, .1, .2 respectively).  I needed to add an id_rsa-style key to the ipp user since ippb04 refuses to use dsa keys.  (copy of existing authorized_keys saved as authorized_keys.20190207)
 
-=== Saturday : 2019.02.09 ===
 
 JRF: Have been checking through the list of cluster nodes using neb-host to find nodes that have been left in repair. So that they can be brought back into production if they are ok 
@@ -358 +353 @@
 }}}
 
+=== Saturday : 2019.02.09 ===
+
+JRF: MEH took TdB and myself through checking out ganlia. Monitoring the general trends on CPU, memory and network loads on various machines. How to spot problems e.g. If the CPU report displays mostly orange/red then it indicates that the machine is often waiting before it can do something, which suggests a more thorough investigation. 
+
+Can check individual machines in pantasks (or add/delete them too).
+pantasks_client
+controller host check ippc118
+
+Configuration failed on processing some files.
+Used the following:
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id
+}}}
+This was to solve some warps that got stuck in the warp stage when reprocessing, the chip im files just needed to be updated. Found out by querying:
+SELECT warp_id, state, label FROM warpRun WHERE label LIKE "ps_ud_WEB";
+then:
+select chip_id,chipRun.state,chipRun.label,chipRun.data_group from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warp_id=1043753;
+Repeated for the the remainder stuck in this state.
+
+
+There are a bunch of wfix things stuck in the cam stage, so queried those to get the chip 
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label PSNSC.wfix.20180525 -chip_id
+{{{
++---------+---------+---------+---------+--------+
+| warp_id | fake_id | cam_id  | chip_id | state  |
++---------+---------+---------+---------+--------+
+| 1043758 | 1037107 | 1061440 | 1093979 | update | 
+| 1168814 | 1161011 | 1185484 | 1210911 | update | 
+| 1168816 | 1161013 | 1185486 | 1210913 | update | 
+| 1169004 | 1161150 | 1185635 | 1211006 | update | 
+| 1169008 | 1161154 | 1185639 | 1211010 | update | 
+| 1169202 | 1161493 | 1185817 | 1211220 | update | 
+| 1169204 | 1161495 | 1185820 | 1211223 | update | 
+| 1175590 | 1168954 | 1193676 | 1228151 | update | 
+| 1181423 | 1173791 | 1198632 | 1233292 | update | 
+| 1187981 | 1179229 | 1203026 | 1238323 | update | 
+| 1265032 | 1258067 | 1284307 | 1321028 | update | 
+| 1265037 | 1258072 | 1284316 | 1321033 | update | 
+| 1842166 | 1835551 | 1863463 | 1895949 | update | 
++---------+---------+---------+---------+--------+
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1093979
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1210911
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1210913
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1211006
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1211010
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1211220
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1211223
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1228151
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1233292
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1238323
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1321028
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1321033
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1895949
+}}}
+
+
+Do this on the MOPS.2 stuff to get them going again:
+chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name 
+o5327g0047o
+o5675g0136o
+o5675g0143o
+o5829g0438o
+o5328g0050o
+Once confirmed that the data has been processed, clean up the old stuff...
 
 === Sunday : 2019.02.10 === 
@@ -363 +422 @@
  * MEH: power at ATRC down, Haydn powered down ippb06-b23 -- set neb-host down for all machines so any primary copies there will switch to secondary on the system 
   * Gene has ippb04,b05 in repair to rsync and retire -- there are now no backup/archive nodes for replication -- NO processing can proceed -- this is not an acceptable situation as discussed before and a backup node set and plan is needed for future ATRC down event.
+
+JRF: ME, MEH, and TdB never got round to going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area). TODO at a later date.