PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.08

• 08:45 MEH: ippmd using ~280 nodes now that nightly is finished (ippsXX, 2x x2)
• 09:05 MEH: WS diffs queued fine today -- will be stdlocal ~300, ippmd~300, stdsci~300 -- except 600 diffs will shortly shut off chip-warp in stdlocal
• 12:55 MEH: the 20T data nodes with space that had put neb-host up late last week to fill and increase the number of data nodes have filled or behaving poorly -- neb-host repair on all 20T nodes now
  • for the 20T nodes w/o processing things seemed mostly okay -- net in was high >50MB/s and seemed to be writing okay, just forced to write constantly and probably could be put back in, but ones doing processing probably shouldn't be

• 20:35 HAF: various emails floating around, there is a problem with nebulous (mark / gene noticed), and summit copy /registration are faulty. The errors we see are like this:

-> pmConfigConvertFilename (pmConfig.c:1833): System error
     failed to create a new nebulous key: nebclient.c:1012 nebSetServerErr() - SOAP-ENV:Server - error: DBD::mysql::st execute failed: The table 'storage_object' is full at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 275,  line 12.
 -> pmConfigRead (pmConfig.c:618): System error
     Unable to resolve trace destination: neb://ipp015.0/gpc1/ThreePi.nt/2014/12/09//o7000g0063o.833489/o7000g0063o.833489.ch.1316586.XY23.trace
Unable to perform ppImage: 1 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/chip_imfile.pl line 830
	main::my_die('Unable to perform ppImage: 1', 833489, 1316586, 'XY23', 1) called at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/chip_imfile.pl line 509

• 20:35 HAF: Serge is investigating, notes that ippdb00 is full and is finding the magical incantations to fix that.
• 20:48 SC: Magic incantation is this: PURGE BINARY LOGS TO 'mysqld-bin.003942';
• 22:17 HAF: seeng the same errors again. this time for 'instance' table. Now what? It's jamming up registration and stuff.

Tuesday : 2014.12.09

• 07:45 EAM : processing has been limping along, with a number of the 'table full' errors. there is enough room on the disk after Serge purged the binary logs, so that is not the cause. the tables are big, but not approaching the InnoDB 64 TB max values. The load on the machine (ippdb00) is modest (3-4). I am guessing that a restart of mysql might clear out something which is cached?

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD