== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : YYYY.MM.DD === === Tuesday : YYYY.MM.DD === === Wednesday : YYYY.MM.DD === * 23:55 MEH: ipp036 down.. take out of processing, neb-host down while try to power cycle -- nothing on console, up but out of processing.. * was already in repair -- leave as such === Thursday : 2014.10.09 === * 10:20 MEH: noticed stdsci underloaded early this morning, needs regular restart (doing summit+reg as well) * will put ipp036 back into processing, so keep an eye on.. * 12:20 MEH: ippsXX in use for MOPS chunk reprocessing test with pre-tag trunk * 14:50 MEH: ipp070 is getting hammered.. set neb-host repair * 17:30 MEH: ippsXX back to lanl stdlocal * 22:00 MEH: nightly processing is getting clobbered, update labels out, turning down lanl stdlocal * ipp082, ipp076, then ipp079 overloaded cycled into/out repair -- something is being excessive * looks like skycell_jpeg.pl running.. how long has this been.. these easily kill processing if too many.. 200 running.. great.. OFF * 22:50 MEH: things slowly clearing and diverting ippsXX to stdsci to catchup.. -- what a great waste of an evening.. === Friday : 2014.10.10 === * 00:55 MEH: net io and data node loads appear higher than normal w/ ippsXX not in use, leaving off and not turning back on in lanl stdlocal for now * 08:00 MEH: ippsXX back on in lanl stdlocal * 13:30 MEH: ippsXX back off in lanl stdlocal -- using for MOPS test processing * 17:40 CZW: restarting stdlocal. Will ensure ippsXX remain off. * 18:15 MEH: new trunk has another issue for mops not quickly solved for testing, returning ippsXX back to stdlocal === Saturday : YYYY.MM.DD === * 07:00 MEH: registration stuck again {{{ regtool -updateprocessedimfile -exp_id 807814 -class_id XY14 -set_state pending_burntool -dbname gpc1 }}} * 19:45 MEH: regular restart stdsci needed === Sunday : YYYY.MM.DD ===