Changes between Version 9 and Version 10 of PS1_IPP_Czarlog_20190304

Timestamp:

Mar 7, 2019, 10:29:49 AM (7 years ago)

Author:

cclin33

Comment:

--

PS1_IPP_Czarlog_20190304

@@ -279 +279 @@
 === Thursday : 2019.03.07 ===
 
+CCL: ipp067 shows I/O issues, waiting for Haydn's help.
+{{{
+Mar  7 03:06:26 ipp067 [57998991.271340] FW state [-268435456] hasn't changed in 180 secs
+Mar  7 03:06:26 ipp067 [57998991.271357] megaraid_sas:adapter not ready
+Mar  7 03:06:27 ipp067 [57998992.279310] megaraid_sas: pending cmds after reset
+Mar  7 03:06:27 ipp067 [57998992.280041] sd 0:2:0:0: rejecting I/O to offline device
+Mar  7 03:06:27 ipp067 [57998992.280058] sd 0:2:0:0: rejecting I/O to offline device
+Mar  7 03:09:46 ipp067 [57999190.870411] XFS (sda1): xfs_log_force: error 5 returned.
+Mar  7 03:09:46 ipp067 [57999190.870432] XFS (sda1): xfs_do_force_shutdown(0x1) called from line 1178 of file fs/xfs/xfs_buf.c.  Return address = 0xffffffff8120a899
+Mar  7 03:09:58 ipp067 [57999202.393163] XFS (sda1): xfs_log_force: error 5 returned.
+Mar  7 03:10:01 ipp067 cron[26168]: (root) CMD (/usr/local/haydn/GetIppDashboardInfo.py)
+Mar  7 03:10:01 ipp067 cron[26169]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
+}}}
+
+CCL: set ipp067 down and off from nebulous.
+{{{ 
+neb-host ipp067 down --note 'CCL: rejecting I/O to offline, so set it down to fix'
+check_system.sh hostoff ipp067 
+}}}
+try a few times "hostoff" because there might be multiple jobs.
+
+CCL: modify the "pantasks_hosts.input" file to exclude listing ipp067
+
+CCL: there are more than 100k njobs and it may cause problems, so we need to restart stdscience to clean all the jobs, buff, etc.
+{{{
+under pantasks_client
+stop
+shutdown now
+}}}
+
+CCL: login to ippc30 to restart ippitc stdscience
+{{{
+under pantasks_client
+server input inpur (this is file name)
+setup (don't do twice!)
+run
+}}}
+
+CCL: one file need to be reprocessed
+{{{
+neb-ls -P neb://ipp092.0/gpc1/20150331/o7112g0341o/o7112g0341o%
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenOSS.20190307 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenOSS.20190307 -set_reduction SWEETSPOT -exp_name o7112g0341o
+under pantasks_client
+add.label
+}}}
+
+
 === Friday : 2019.03.08 ===