PS1 IPP Czar Logs for the week 2011-03-28 - 2011-04-03

(Up to PS1 IPP Czar Logs)

Monday: 2011-03-28

• (serge) Started gpc1 ingestion for replication on ippc02 (at 09:52:21)
• 14:00 CZW: Stopped all pantasks to allow for upgrade of ISP database.
• 14:39 CZW: Beginning flush for binlog status and initial dump.
• 14:47:31 gpc1 ingestion complete
• 14:57 gpc1 replication started. Seconds_Behind_Master: 327235
• 14:58 CZW: Finished updates to ISP database. Beginning flush for binlog status for final dump.
• 15:04 CZW: Making final dump. Completed while typing. Upgrade complete. See attached logfile for details.
• 22:44 CZW: ippdb01 had load of ~20. Stopped all pantasks, let jobs finish, and then checked mysql processlist. Even with all jobs finished or timedout, a large number of queries were still pending. Cleared them, and restarted summitcopy and registration. Removed old dates from stdscience pantasks, as it might be adding extra queries for completed dates (some dates did not have morning darks taken).
• 22:47 CZW: restarting stdscience.
• 22:50 CZW: removed old dates from registration to ensure burntool backlog can be cleared without waiting for the date to cycle around (seven old dates were still being examined).
• 23:14 CZW: registration promoted more things to stdscience, and task timeouts again started at fake.imfile.load. Registration/summitcopy appears to be fully caught up, so I don't believe that that is adding a significant load on the database. Without much else to do, turning stdscience back on. Perhaps a shutdown/restart would resolve things?
• 23:22 CZW: continuing problems with fake.imfile.load, stopped, waiting for jobs to clear, then will restart stdscience.
• 23:35 CZW: Removed labels we're not processing to see if
• 23:46 CZW: Looking at the slow fake_pendingimfile run, it looks like the issue may just be a set of bad joins that slows the query down. Why this is suddenly an issue is something I don't know.
• 00:12 CZW: Rewrote fake_pendingimfile.sql JOINS to work in a more efficient manner.
• 00:23 CZW: Restarted remaining pantasks using new fake_pendingimfile.sql. Nothing seems to have exploded yet.
• 01:06 CZW: Reverting pzDownloadImfiles that seem to be clogging up registration. Doesn't seem to be effective, probably a result of ipp050 being down. It should all continue successfully once ipp050 comes back up.
• 01:17 CZW: Tired of waiting, moved spurious nebulous keys to $key.bak, and reverted the summit copies. This appears to have caught, and we're registering things again.

Tuesday: 2011-03-29

Wednesday: 2011-03-30

Thursday: 2011-03-31

Friday: 2011-04-01

Saturday: 2011-04-02

Sunday: 2011-04-03