== PS1 IPP Czar Logs for the week 2015.03.16 - 2015.03.22 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.03.16 ===
 * 07:06 Bill: postage stamp server was a bit sluggish. pcontrol spinning so restarted it. 
  * Cleared some MPIA request faults due to problems at their data store. I need to start reverting fault 200 (HTTP 500) which their data store installation gives from time to time.
  * lowered QUB's priority to be even with MPIA and MPG since they have a large backlog of requests requiring update processing.

=== Tuesday : 2015.03.17 ===
 * 06:00 EAM: stdscience running a bit slow, restarting; also restarting pv3diff
 * 10:21 Bill: disabled  ipp-misc site on ippc17 which is where the getsmf script lives. It requires nebulous.
 * 17:20 EAM: Bill upgraded the ippdb00 mysql to 5.6 today.  things restarted fine and seemed to run ok for a while.  around 16:15, Bill reported errors going to the log.  looking into these, I concluded we were having trouble with locks due to the innodb adaptive index hash (see http://dev.mysql.com/doc/refman/5.6/en/innodb-adaptive-hash.html).  I turned of adaptive hash with the following:
{{{
mysql> set global innodb_adaptive_hash_index = 0;
}}}
So far, it looks like this is helping (we are not getting a build up of long thread waits in the SEMAPHORES section of show engine innodb status\G)

=== Wednesday : 2015.03.18 ===

=== Thursday : 2015.03.19 ===

=== Friday : 2015.03.20 ===

=== Saturday : 2015.03.21 ===

=== Sunday : 2015.03.22 ===