IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 6 and Version 7 of PS1_IPP_Czarlog_20170731


Ignore:
Timestamp:
Aug 4, 2017, 5:13:44 PM (9 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20170731

    v6 v7  
    3232[ 1411.290072]  <<EOE>> <4>---[ end trace 9eaaeb060d55ea58 ]---
    3333}}}
     34
    3435=== Wednesday : 2017-08-02 ===
    3536 * 15:00 CZW: Adding x3 to the /data/stare04.1/watersc1/hsc/stdsci2/ pantasks, in an attempt to get HSC stacks finished at a reasonable rate.  With stare03/04 now responding properly to NFS, this should not cause mount issues.
    36 === Thursday : YYYY.MM.DD ===
    3737
     38=== Thursday : 2017.08.03 ===
     39 * MEH: cleared stalled/broken stamps from 7/30, unclear why weren't cleared by czar..
     40 * MEH: HVAC planned downtime 8/4-8/6 @IfA -- Mez computers that are not critical for operations will need to be powered down
     41  * shutdown any mysql, halt and power down ipp003,ipp001,ipp022,ippb04,ippb05; ippc20,c21,c22,013,018-021 (Gavin powered down and manually off since not on remote power control)
     42  * monitor temperatures with sensors on ippops1,ippops2,ipp002
     43 * MEH: Haydn sent Chris email about working on console for ipp074,075,076,114 -- appears to actually not be working... -- Haydn had turned off i04-con (impacts access to ippdb, ipp105-117 nodes critical for nightly and PSPS processing), Ming turned back on, but still critical problem
     44  * 086-089,091,092 remote power management okay but no serial console access -- leave neb-host repair since minimal data space
     45  * 105-112 no remote power management now.. -- critical problem -- put ipp105 into neb-host repair to avoid breaking nightly processing if node has problem (one of the primary nightly data nodes)..
     46 * MEH: ipp100-104 neb-host up to add extra data space for ipp105 into repair
     47 * MEH: more misc cleanup of SNIaF updates
    3848 * 17:25 CZW: Power cycling stare01, which crashed again.
    3949{{{
     
    96106[115569.810018] ---[ end trace c9a61e19b0bbc266 ]---
    97107}}}
    98 === Friday : YYYY.MM.DD ===
     108 * MEH: nightly processing issue in summitcopy, 3-5 minutes to download some otas -- from the timestamps in summit.copy.log, appears delay is happening in copy into nebulous stage... seemed to also be the case last night as well..
     109
     110=== Friday : 2017.08.04 ===
     111 * MEH: reconfigure ippitc targeting to include ipp100-104 tonight
    99112
    100113=== Saturday : YYYY.MM.DD ===