Changes between Version 15 and Version 16 of PS1_IPP_Czarlog_20160404

Timestamp:

Apr 8, 2016, 5:45:50 PM (10 years ago)

Author:

watersc1

Comment:

--

PS1_IPP_Czarlog_20160404

@@ -29 +29 @@
 === Friday : 2016-04-08 ===
  * 15:40 CZW: Restarting ipp/pantasks.
+ * 17:35 CZW: ippc05 is having problems.
+  {{{This host runs the ipp/cleanup pantasks.  I noticed that after the
+restart of pantasks, it wasn't completing any of those jobs
+(lossycomp).  Logging in resulted in a lot of odd hanging.  A dmesg
+suggests a disk failure:
+[...lots of these...]
+[16951939.150146] ata2.00: status: { DRDY ERR }
+[16951939.150148] ata2.00: error: { UNC }
+[16951939.151880] ata2.00: configured for UDMA/133
+[16951939.151896] ata2: EH complete
+[16951946.152075] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
+[16951946.152079] ata2.00: irq_stat 0x40000001
+[16951946.152084] ata2.00: cmd c8/00:08:f9:43:73/00:00:00:00:00/e3 tag 0 dma 4096 in
+[16951946.152085]          res 51/40:08:f9:43:73/00:00:03:00:00/e3 Emask 0x9 (media error)
+[16951946.152087] ata2.00: status: { DRDY ERR }
+[16951946.152089] ata2.00: error: { UNC }
+[16951946.153872] ata2.00: configured for UDMA/133
+[16951946.153892] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
+[16951946.153895] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
+[16951946.153900] Descriptor sense data with sense descriptors (in hex):
+[16951946.153902]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
+[16951946.153911]         03 73 43 f9
+[16951946.153914] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
+[16951946.153919] end_request: I/O error, dev sdb, sector 57885689
+[16951946.153945] ata2: EH complete
+[16951946.153991] sd 1:0:0:0: [sdb] Write Protect is off
+[16951946.153995] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
+[16951946.154170] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
+[16951946.154207] sd 1:0:0:0: [sdb] 976773168 512-byte hardware sectors: (500 GB/465 GiB)
+[16951946.154222] sd 1:0:0:0: [sdb] Write Protect is off
+[16951946.154224] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
+[16951946.154425] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
+[16951946.168432] raid1:md2: read error corrected (8 sectors at 25690376 on sdb3)
+[16951946.168447] raid1: sda3: redirecting sector 25690376 to another mirror
+
+I've stopped and shutdown the cleanup pantasks, and will restart it on
+a different host (ippc11) for the weekend.  I've also removed ippc05
+from the nebulous server list for the ipp and ippqub users.
+}}}
 === Saturday : YYYY.MM.DD ===