== PS1 IPP Czar Logs for the week 2013.04.29 - 2013.05.05 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday :2013.04.29 === mark is czar * 07:00 MEH: lookslike 63 exposures stuck in registration, auto fixing of check_burntool problem not working now? * 14:50 MEH: stsci nodes gone, abducted by aliens? no can reach by console so must be network issue. may want to do neb-host down if gone for a while -- Gavin was working on host config issue, all okay. * 17:00 MEH: doing regular restart of stdscience === Tuesday : 2013.04.30 === mark is czar * 04:00 MEH: manually reverted some 40 stacks (ESS,MD), normal nfs faults seem to be happening more often now. auto-revert needed.. looks like typical random compute nodes having issue, maybe overloaded. last night and part of the SAS test stacks Chris ran were 90% from ippc40 in the stack pantask log (nothing clear in messages why and mounts okay). * 11:10 MEH: enabling stack.revert -- removing all non-nightly science labels from ~ipp/stack/input, turned off in stack.pro task itself but adding stack.revert.on into the input file === Wednesday : 2013.05.01 === * 13:30 Bill: stopped all processing and apache servers on ippc17 (datastore) and ipp049 (pstamp-test) to rebuild the ippRequestServer database on ippc17 using innodb tables and restore replication to ippc19 * 16:00 started up summit copy and registration === Thursday : 2013.05.02 === Bill is czar today * 04:00 EAM : reconfigured mysql on ipp005 - ipp009 : set innodb_buffer_pool_size back down to 2G from 16G. I had bumped these up in an attempt to help ipptopsps / dvopsps go faster (avoid swapping), but have since switched to memory engine for the dvoDetectionFull table causing the problem. I also killed some rpc.statds on ipp008 and ipp010. * 05:12 replication of ippRequestServer database from ippc17 to ippc19 has failed. The error is that an insert into the slave failed because an insert into dsFileset tried to use a value for the primary key that duplicates an existing row {{{ On the slave mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: ippc17.ifa.hawaii.edu Master_User: repl_pstamp Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysqld-bin.000718 Read_Master_Log_Pos: 63235322 Relay_Log_File: mysqld-relay-bin.000002 Relay_Log_Pos: 1751685 Relay_Master_Log_File: mysqld-bin.000718 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: ippRequestServer Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '3478690' for key 1' on query. Default database: 'ippRequestServer'. Query: 'INSERT into dsFileset (prod_id, fileset_name, reg_time, type, prod_col_0, prod_col_1, prod_col_2, prod_col_3, prod_col_4, prod_col_5, prod_col_6, prod_col_7) VALUES('52', 'dqstats.20794', UTC_TIMESTAMP(), 'table', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)' Skip_Counter: 0 Exec_Master_Log_Pos: 1751547 Relay_Log_Space: 63235460 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL 1 row in set (0.00 sec) On the master the troublesome row is select * From dsFileset where fileset_id = 3478690; +---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+ | prod_id | fileset_id | fileset_name | reg_time | hide | type | prod_col_0 | prod_col_1 | prod_col_2 | prod_col_3 | prod_col_4 | prod_col_5 | prod_col_6 | prod_col_7 | +---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+ | 52 | 3478690 | dqstats.20794 | 2013-05-02 06:18:09 | 0 | table | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | +---------+------------+---------------+---------------------+------+-------+------------+------------+------------+------------+------------+------------+------------+------------+ 1 row in set (0.00 sec) While on the slave that id is used for another entry mysql> select * From dsFileset where fileset_id = 3478690; +---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+ | prod_id | fileset_id | fileset_name | reg_time | hide | type | prod_col_0 | prod_col_1 | prod_col_2 | prod_col_3 | prod_col_4 | prod_col_5 | prod_col_6 | prod_col_7 | +---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+ | 54 | 3478690 | o6414g0056o.camera.808365.2945979.26 | 2013-05-02 06:17:09 | 0 | IPP-DIST | 1316 | camera | 808365 | o6414g0056o | ThreePi.20130502 | y.00000 | NULL | NULL | +---------+------------+--------------------------------------+---------------------+------+----------+------------+------------+------------+-------------+------------------+------------+------------+------------+ 1 row in set (0.00 sec) But that entry does not exist on the master mysql> select * From dsFileset where fileset_name = 'o6414g0056o.camera.808365.2945979.26'; Empty set (0.50 sec) Interestingly the corresponding fileset directory does exist in the dsroot directory on ippc17. bills% ls /data/ippc17.0/datastore/dsroot/ps1-3pi-cat/o6414g0056o.camera.808365.2945979.26 dbinfo.camera.808365.mdc index.txt dirinfo.camera.808365.mdc o6414g0056o.607641.cm.808365.tgz But is inacessible through the datastore because the wcat http://ippc17/ds/ps1-3pi-cat/o6414g0056o.camera.808365.2945979.26/index.txt failed to find o6414g0056o.camera.808365.2945979.26 in ps1-3pi-cat }}} * 05:56 It appears that the rcserver task in distribution pantasks has been inserting data directly into the slave ippc19. This is very strange === Friday : 2013.05.03 === === Saturday : YYYY.MM.DD === === Sunday : YYYY.MM.DD ===