PS1 IPP Czar Logs for the week 2017.04.17 - 2017.04.23

(Up to PS1 IPP Czar Logs)

Monday : 2017.04.17

Nebulous/pantasks was turned off in order for the copy (db01) to be rsynced somewhere safe. It is still ongoing, we anticipate it will be done tomorrow morning. HAF will check status and restart when it is complete. No pantasks for tonight. Nightly processing will start tomorrow morning (if all goes well).

Tuesday : 2017.04.18

• MEH: Serge is running transfer of nebulous mysql files from ipp122->ipp115 -- ipp115 will have high load until finished, nothing else should be using ipp115/ipp122
  • ippdb01 (master) and ipp115 (slave) setup and active again -- restart all processing, stamps+stdscience updates seem fine, summitcopy+registration darks ok, cleanup ok
  • weather clearing so allowing nightly to process and leaving things running -- rate is lower than normal, chips seems to get stuck 400-600s at times

Wednesday : 2017.04.19

• Weryk using ippc76-ippc126 for processing during day per Gene's comment at the group meeting
• HAF restarted pantasks at 17:30

Thursday : 2017.04.20

• MEH: ippdb06 nebulous replication rsync -- disk writing rate much lower than expected, Haydn triggered raid initialization (won't wipe but need to suspend transfer so it can finish faster) -- still seems to have a reduced disk write rate ~50-60MB/s
  • rebuild mysql 5.6 for use on new raid and modify configs to use ippdb06.1 instead
• MEH: ipp121 back online after Haydn and Gavin swapped out disks and restored the OS -- /etc/exports empty, copied and modified ipp122 version; key problem, cleared for ipp,ippmops,ippqub -- leave in repair for a while (probably should until remote power access is setup)

Friday : 2017.04.21

Saturday : 2017.04.22

• MEH: ippc111 appears to have crashed ~1700 -- taking out of processing and comment out in ~ippitc/ippconfig/pantasks_hosts.input

Sunday : 2017.04.23