== PS1 IPP Czar Logs for the week 2016.07.18 - 2016.07.24 ==
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.07.18 ===
 * 17:30 MEH: updating handful of images and making new diffims for MOPS w/ label like OSS.P10vX3a 
 * 23:10 MEH: QUB.lowdec.20160712 stacks running for week on ippsXX nodes (data targeted to ipp100-104), pantasks on ipps00 ~ippqub/stack if needs to be stopped
=== Tuesday : 2016.07.19 ===
 * 09:00 MEH: updating images for MOPS
 * 09:35 MEH: lots of pantasks logs haven't been regularly archived (see [wiki:Processing]) -- running now to free up wasted space on ippc19... ~12GB
 * 10:40 MEH: ipp063 disk replaced and raid rebuilt 7/18, neb-host up again
 * 10:45 MEH: ipp078, 080 had SSD issues, replaced 5/12... should finally be put back up and checked... seriously...
 * 10:50 MEH: start setting regularly red nodes to repair to preserve space for ipptopsps -- ipp070,071,079,083-096
 * 11:25 MEH: Gene reports Haydn reports that the spontaneous reboot nodes ipp008,012,013,014,016,018 in cab1 shouldn't suffer from brownouts now that ipp037 has been decom and removed -- so set neb-host up
 * 11:35 MEH: ipp017 was reconstructed from decom ipp033 on 5/3... also neb-host up
 * 17:10 MEH: ipp101 up from repair now.. apparently someone left all the directories owned by ipp not apache and was preventing nebulous from writing files (reported problem killing nightly processing [wiki:PS1_IPP_Czarlog_20160516#Monday:2016.05.16] and in email)...
 * 21:00 MEH: sending older QUB and ps_ud% data to cleanup, leaving the .multi for Chris to deal with
=== Wednesday : 2016.07.20 ===

=== Thursday : 2016.07.21 ===
 * 14:00 MEH: QUB.lowdec.20160712 stacks temporarily running on upper ippx nodes not in use (049-088, excluding 061) -- can remove from nodes when needed
 * 16:55 MEH: looks like ippx071 was never setup to do IPP processing properly... no ipp dir, no tmp dir etc... setting up now..
=== Friday : 2016.07.22 ===

=== Saturday : 2016.07.23 ===

=== Sunday : 2016.07.24 ===