Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20190128

Timestamp:

Feb 2, 2019, 7:06:46 AM (7 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20190128

@@ -20 +20 @@
  * TdB: ippb16-20 permissions for nebulous drives fixed to apache, and Gene is running chown on all files on these machines. Will be set back to up on monday, due to Friday being a bad day for system-wise changes.
  * TdB: machine ipp70 removed from /home/panstarrs/ipp/local/bin/apachedisk_chk.sh since it is down (and had already been removed from nebulous), but still spawning warning emails every 4 hours.
- * MEH: something caused >250k warps in various update states saved for MOPS and QUB (ps_ud_MOPS%/QUB%) to get send to cleanup what looks to be ~noon, but didn't notice until nagios sent load warning for ippdb09 ~1523 -- all warps are being retained by default, only broken or test ones should be cleaned
+ * MEH: something caused >250k warps in various update states saved for MOPS and QUB (ps_ud_MOPS%/QUB%) to get send to cleanup what looks to be ~noon, but didn't notice until nagios sent load warning for ippdb09 ~1523 -- all warps are being retained by default, only broken or test ones should be cleaned -- a large number of cleanups will take a long time to get through, while in that state it the more there are the more chance it will block+fault pstamp requests wanting to update part of that warp 
   * stop cleanup pantasks and manually try to salvage remaining possible warp_id set state update and label update_recovery for all state goto_cleaned with data_state full
+{{{
+warptool -dbname gpc1 -updaterun -state goto_cleaned -label goto_cleaned -set_state update -set_label update_recovery -warp_id XXXX
+}}}
 
 === Saturday : 2019.02.02 ===
- * MEH: salvage of warps finished, cleanup back to run
+ * MEH: salvage of warps finished, cleanup back to run -- some are broken (several cycles of fail in cleanup), change label
+{{{
+warptool -dbname gpc1 -updaterun -state goto_cleaned -label goto_cleaned -set_label goto_cleaned.broken
+}}}
+
 === Sunday : YYYY.MM.DD ===