PS1 IPP Czar Logs for the week 2013.04.29 - 2013.05.05

(Up to PS1 IPP Czar Logs)

Monday :2013.04.29

mark is czar

• 07:00 MEH: lookslike 63 exposures stuck in registration, auto fixing of check_burntool problem not working now?
• 14:50 MEH: stsci nodes gone, abducted by aliens? no can reach by console so must be network issue. may want to do neb-host down if gone for a while -- Gavin was working on host config issue, all okay.
• 17:00 MEH: doing regular restart of stdscience

Tuesday : 2013.04.30

mark is czar

• 04:00 MEH: manually reverted some 40 stacks (ESS,MD), normal nfs faults seem to be happening more often now. auto-revert needed.. looks like typical random compute nodes having issue, maybe overloaded. last night and part of the SAS test stacks Chris ran were 90% from ippc40 in the stack pantask log (nothing clear in messages why and mounts okay).
• 11:10 MEH: enabling stack.revert -- removing all non-nightly science labels from ~ipp/stack/input, turned off in stack.pro task itself but adding stack.revert.on into the input file

Wednesday : 2013.05.01

• 13:30 Bill: stopped all processing and apache servers on ippc17 (datastore) and ipp049 (pstamp-test) to rebuild the ippRequestServer database on ippc17 using innodb tables and restore replication to ippc19
• 16:00 started up summit copy and registration

Thursday : 2013.05.02

Bill is czar today

• 04:00 EAM : reconfigured mysql on ipp005 - ipp009 : set innodb_buffer_pool_size back down to 2G from 16G. I had bumped these up in an attempt to help ipptopsps / dvopsps go faster (avoid swapping), but have since switched to memory engine for the dvoDetectionFull table causing the problem. I also killed some rpc.statds on ipp008 and ipp010.
• 05:12 replication of ippRequestServer database from ippc17 to ippc19 has failed. The error is that an insert into the slave failed because an insert into dsFileset tried to use a value for the primary key that duplicates an existing row

On the master
select * From dsFileset where fileset_id = 3478690;
+---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+
| prod_id | fileset_id | fileset_name  | reg_time            | hide | type  | prod_col_0 | prod_col_1 | prod_col_2 | prod_col_3 | prod_col_4 | prod_col_5 | prod_col_6 | prod_col_7 |
+---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+
|      52 |    3478690 | dqstats.20794 | 2013-05-02 06:18:09 |    0 | table | NULL       | NULL       | NULL       | NULL       | NULL       | NULL       | NULL       | NULL       | 
+---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+
1 row in set (0.00 sec)

On the slave
mysql> select * From dsFileset where fileset_id = 3478690;
+---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+
| prod_id | fileset_id | fileset_name                         | reg_time            | hide | type     | prod_col_0 | prod_col_1 | prod_col_2 | prod_col_3  | prod_col_4       | prod_col_5 | prod_col_6 | prod_col_7 |
+---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+
|      54 |    3478690 | o6414g0056o.camera.808365.2945979.26 | 2013-05-02 06:17:09 |    0 | IPP-DIST | 1316       | camera     | 808365     | o6414g0056o | ThreePi.20130502 | y.00000    | NULL       | NULL       | 
+---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+
1 row in set (0.00 sec)

But this fileset is missing on the master
mysql> select * From dsFileset where fileset_name = 'o6414g0056o.camera.808365.2945979.26';
Empty set (0.50 sec)

Friday : 2013.05.03

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD