== PS1 IPP Czar Logs for the week 2016.10.31 - 2016.11.06 ==
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.10.31 ===
 * MEH: NCU/MOPS processing today, upper ippx and ipps nodes use continues
 * MEH: ippMonitor has had ippdb03 red for a while -- password for ippMonitor needed to be fixed in config -- showing slave status properly now
 * MEH: relastro reported done at meeting -- should reallocated nightly nodes back to default (x0,1->s3,4) since Heather using for next loading processed
  * oddly seems s6 also commented out -- #  EAM 2016.05.04 : remove s6 nodes to test load for ITS move -- is this still necessary? -- no, was a short time test and the 3x should have been put back into processing
=== Tuesday : 2016.11.01 ===
 * MEH: NCU/MOPS processing mostly finished except for ipps node use (and normal nightly c2 node use) -- waiting for request for products as needed now
 * 13:20 MEH: Haydn taking one of datanodes on 10G offline to test something with the cable? setting processing to stop -- finished and restarted all pantasks for nightly processing
  * many of nightly datanodes are red... cleaning some of the NCU diffs to make space since appears not any other space to be freed
=== Wednesday : 2016.11.02 ===

 * 9:00 CZW: Processing stopped to allow for switchover to N5K switch for ippdb01.
 * 12:45 CZW: Processing resumed.
 * 16:25 CZW: Pantasks restart, even if we didn't have data last night.
 * MEH: QUB updating modest amount of NCU diffs for stamps off and on possibly (regularly part of ~ippqub/stdscience_ws and independent of nightly processing for MOPS)
 * MEH: SNIaF updates running on ipps nodes in ~ippmops/stdscience
=== Thursday : 2016.11.03 ===
 * MEH: reconfig for nightly QUB targeted processing -- finished until after midnight possibly tonight
 * 16:30 CZW: Restart of pantasks after rebuild of ippTools.  This picks up the change to prioritize the download of image files that do not have an exp_name LIKE 'c%' above those that do match that exp_name.
 * MEH: cleaning up remaining NCU WSdiff to hopefully free up more space for normal nightly data..
 * MEH: pstamp was stalled on MOPS request for >hr,  restart cleared issue
=== Friday : 2016.11.04 ===

=== Saturday : 2016.11.05 ===

=== Sunday : 2016.11.06 ===