PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014-02-10

• 09:40 Heather stopping loaders
• 09:45 MEH: LAP label out of stdsci to try and catchup last nights data. ThreePi.WS.nightlyscience out as well
• 09:50 MEH: still seeing loader machines having trouble, putting neb-host repair and taking out of processing as it holds up summit+registration -- ipp043,048--062 many can probably be brought back up one at a time during LAP once things settle down
• 10:35 MEH: see MOPS stamps showing up, adding c3 to pstamp to help speed those along (should run fine with other normal processing jobs there, no ghost jobs...)
• 10:50 MEH: turning hosts on in summitcopy that are normally off to try and boost the downloads since many wave4 had to be turned off from fallout from loaders. will want those turned off afterwards unless monitored as they are untested for reliability.
• 11:15 CZW: A check of the summitcopy pantasks.stdout.log shows that all failures this morning are "Nebulous::Client::stat - unhandled fault - database error: no storage object found" for ota42. This points to an issue with ipp017, so I have set that to repair in nebulous.
• 11:30 MEH: ipp054 is in a bad state, probably needs to be rebooted -- so out of processing and in neb-host repair

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD