Changes between Version 17 and Version 18 of PS1_IPP_Czarlog_20190510

Timestamp:

May 14, 2019, 11:26:46 AM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -91 +91 @@
  --------------------
 
- There are also some I/O relatd errors with ippb18, with fault codes 3 and 4. Judging by Ganglia, it is seeing some fairly heavy usage. Send the exps to revert, but put ippb18 into repair for the time being to give it a break.
+ There are also some I/O related errors with ippb18, with fault codes 3 and 4. Judging by Ganglia, it is seeing some fairly heavy usage. Send the exps to revert, but put ippb18 into repair for the time being to give it a break.
 {{{
 neb-host --state repair --host ippb18 --note 'TdB: up->repair, due to it getting hammered and producing I/O errors'
@@ -221 +221 @@
 }}}
  The vast majority of them are in fact exps which have a label of goto_clean% which are likely previously cleaned chip exps. However, we cannot simply set their state to cleaned, since we have no way of telling whether the cleanup operation went succesfully or failed. Therefore, we need to send the whole lot back to cleanup. Nonetheless, by parsin the xps, we can preferentially send exps that we still need to use in the update processing before doing things previosuly cleaned. Focus on labels like %LAP.PV3% or %OSS% at first and slowly end them to cleanup in between other cleanup jobs.
+
+ I made a quick inventory of the predominant labels that have large (>100) numbers of wait state chip exps:
+{{{
++--------+-----------------------------------+-----------------------------------+
+| waitct | data_group                        | label                             |
++--------+-----------------------------------+-----------------------------------+
+|    104 | EUCLID.MD07.PV0.T1                | EUCLID.MD07.PV0.T1                | 
+|    104 | EUCLID.MD08.PV0.T1                | EUCLID.MD08.PV0.T1                | 
+|    106 | NULL                              | NULL                              | 
+|    106 | altaz.20080920                    | altaz.20080920                    | 
+|    109 | LAP.PV3.20140730.20140918         | goto_cleaned.broke190323          | 
+|    114 | dropped                           | STS.rp.2013.hold                  | 
+|    117 | scitest.200807                    | scitest.200807                    | 
+|    122 | QUB.20160517                      | QUB.PS16cgx                       | 
+|    141 | flatcorr.20081109                 | flatcorr.20081109                 | 
+|    141 | flatcorr.20100124                 | flatcorr.20100124                 | 
+|    142 | flattest.20091010                 | flattest.20091010                 | 
+|    142 | MD03.20091228                     | MD03.nightlystack                 | 
+|    142 | flatcorr.20091003                 | flatcorr.20091003                 | 
+|    180 | PV2.PHOTO.STD.20141028.i          | PV2.PHOTO.STD.20141028            | 
+|    180 | flatcorr.20081107                 | flatcorr.20081107                 | 
+|    187 | Bright3Pi.20190402                | Bright3Pi.nightlyscience          | 
+|    203 | flatcorr.20090509                 | flatcorr.20090509                 | 
+|    205 | MD07.20091123                     | MD07.20091123                     | 
+|    209 | OSS.20141207                      | goto_cleaned.broke190312          | 
+|    214 | opposition.20080922               | cleanup_testing_old               | 
+|    217 | ThreePi.Run2.z.v0                 | ThreePi.Run2.z.v0                 | 
+|    231 | mops.fixbrokenLAP.20180323        | mopsfix                           | 
+|    248 | SAS.20151208                      | SAS.20151208                      | 
+|    248 | czw.SAS.201402114                 | lanl.SAS.201402114                | 
+|    264 | ThreePi.Run2.i.v0                 | ThreePi.Run2.i.v0                 | 
+|    282 | ThreePi.yz.20091203               | ThreePi.yz.20091203               | 
+|    304 | m31.ps1.20170627                  | m31.ps1.20170627.wait             | 
+|    365 | ThreePi_NorthernRegion.090729     | ThreePi_NorthernRegion.090729     | 
+|    573 | altaz.20080920                    | goto_purged                       | 
+|    596 | LAP.PV3.20140730.20141008         | goto_cleaned.broke                | 
+|    641 | scitest.20090327                  | scitest.20090327                  | 
+|    788 | LAP.PV3.20140730.20150117         | update.LAP.PV3                    | 
+|    967 | LAP.PV3.20140730.20140927         | goto_cleaned.redoitchold          | 
+|   1252 | m31.ps1.20170627                  | m31.ps1.20170627                  | 
+|   1756 | LAP.PV3.20140730.20140807         | PV3.cleanup                       | 
+|   2239 | ThreePi.20140126                  | goto_cleaned.redoitc              | 
+|   3360 | PV3.Pole.Reprocess.20180510       | PV3.Pole.Reprocess.20180510       | 
+|  15431 | STS.rp.2013.20120412              | STS.rp.2013                       | 
+|  51293 | ThreePi.20100220                  | goto_cleaned.rerun                | 
+| 166252 | scitest.200808                    | goto_cleaned_redo                 | 
+| 443284 | altstrip.200806                   | goto_cleaned                      | 
++--------+-----------------------------------+-----------------------------------+
+}}}
+
+
 
  * TdB: at around 17:30, the IPPmonitor stalled. Turns out, mysql crashed on ippdb05. Let's check things: