PS1 IPP Czar Logs for the week 2016.07.18 - 2016.07.24

(Up to PS1 IPP Czar Logs)

Monday : 2016.07.18

• 17:30 MEH: updating handful of images and making new diffims for MOPS w/ label like OSS.P10vX3a
• 23:10 MEH: QUB.lowdec.20160712 stacks running for week on ippsXX nodes (data targeted to ipp100-104), pantasks on ipps00 ~ippqub/stack if needs to be stopped

Tuesday : 2016.07.19

• 09:00 MEH: updating images for MOPS
• 09:35 MEH: lots of pantasks logs haven't been regularly archived (see Processing) -- running now to free up wasted space on ippc19... ~12GB
• 10:40 MEH: ipp063 disk replaced and raid rebuilt 7/18, neb-host up again
• 10:45 MEH: ipp078, 080 had SSD issues, replaced 5/12... should finally be put back up and checked... seriously...
• 10:50 MEH: start setting regularly red nodes to repair to preserve space for ipptopsps -- ipp070,071,079,083-096
• 11:25 MEH: Gene reports Haydn reports that the spontaneous reboot nodes ipp008,012,013,014,016,018 in cab1 shouldn't suffer from brownouts now that ipp037 has been decom and removed -- so set neb-host up
• 11:35 MEH: ipp017 was reconstructed from decom ipp033 on 5/3... also neb-host up
• 17:10 MEH: ipp101 up from repair now.. apparently someone left all the directories owned by ipp not apache and was preventing nebulous from writing files (reported problem killing nightly processing PS1_IPP_Czarlog_20160516 and in email)...
• 21:00 MEH: sending older QUB and ps_ud% data to cleanup, leaving the .multi for Chris to deal with

Wednesday : 2016.07.20

Thursday : 2016.07.21

• 14:00 MEH: QUB.lowdec.20160712 stacks temporarily running on upper ippx nodes not in use (049-088, excluding 061) -- can remove from nodes when needed
• 16:55 MEH: looks like ippx071 was never setup to do IPP processing properly... no ipp dir, no tmp dir etc... setting up now..

Friday : 2016.07.22

Saturday : 2016.07.23

Sunday : 2016.07.24