PS1 IPP Czar Logs for the week 2015.03.16 - 2015.03.22

(Up to PS1 IPP Czar Logs)

Monday : 2015.03.16

• 07:06 Bill: postage stamp server was a bit sluggish. pcontrol spinning so restarted it.
  • Cleared some MPIA request faults due to problems at their data store. I need to start reverting fault 200 (HTTP 500) which their data store installation gives from time to time.
  • lowered QUB's priority to be even with MPIA and MPG since they have a large backlog of requests requiring update processing.

Tuesday : 2015.03.17

• 06:00 EAM: stdscience running a bit slow, restarting; also restarting pv3diff
• 10:21 Bill: disabled ipp-misc site on ippc17 which is where the getsmf script lives. It requires nebulous.
• 17:20 EAM: Bill upgraded the ippdb00 mysql to 5.6 today. things restarted fine and seemed to run ok for a while. around 16:15, Bill reported errors going to the log. looking into these, I concluded we were having trouble with locks due to the innodb adaptive index hash (see ​http://dev.mysql.com/doc/refman/5.6/en/innodb-adaptive-hash.html). I turned of adaptive hash with the following:

  mysql> set global innodb_adaptive_hash_index = 0;
  

So far, it looks like this is helping (we are not getting a build up of long thread waits in the SEMAPHORES section of show engine innodb status\G)

Wednesday : 2015.03.18

Thursday : 2015.03.19

Friday : 2015.03.20

Saturday : 2015.03.21

Sunday : 2015.03.22