IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 10 and Version 11 of PS1_IPP_Czarlog_20190204


Ignore:
Timestamp:
Feb 6, 2019, 4:22:00 PM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20190204

    v10 v11  
    8484
    8585=== Wednesday : 2019.02.06 ===
    86  * MEH: nebulous apache server (ippc70-c75) log reset --
     86
     87JRF: MEH, TdB and myself are going to archive the nebulous logs as they are getting very large, hence nagios complaining about lack of disk space on root. Log sizes:
     88 * ippc71 - 20GB
     89 * ippc72 - 46GB
     90 * ippc73 - 78GB
     91 * ippc74 - 74GB
     92 * ippc75 - 78GB
     93
     94There are also other logs in the /var/log/ folder that need to be archived periodically like messages and apache2
     95
     96 * archive the neb log (in this case this is for ippc75, change the location appropriately)
     97{{{
     98 user@ippc75$ rsync -avP /tmp/nebulous_server.log /export/ippc75.0/
     99}}}
     100
     101* MEH: nebulous apache server (ippc70-c75) log reset --
    87102{{{
    88103
     
    97112sudo /etc/init.d/apache2 status
    98113}}}
     114The above makes sure to check that apache is no longer writing to the log before moving it. After moving a new file needs to be created, with the correct permissions, for apache to write to in the future. Then the service can be restarted.
    99115
    100 JRF: ME, MEH, and TdB shall be going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area).
    101 
    102 JRF: MEH, TdB and myself are going archiving the apaches logs on ippc71-ippc75 as they are gettings large:
    103 ippc71 - log was 20GB
    104 ippc72 - 46
    105 ippc73 - 78
    106 ippc74 - 74
    107 ippc75 - 78GB
    108 
    109  * archive the neb log
    110 {{{
    111 rsync -avP /tmp/nebulous_server.log /export/ippc75.0/
    112 }}}
    113116 * do a test exposure
     117After setting everything back up it's a good idea to do a test exposure. So turn on stdscience from the pantasks_client and run the following chiptool command (change the date).
    114118{{{
    115119chiptool -dbname gpc1 -definebyquery -set_label MOPS.dailytestset -set_workdir neb://@HOST@.0/gpc1/MOPS.dailytestset.20190206 -set_data_group MOPS.dailytestset.20190206 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -simple -exp_name o6370g0351o  -pretend
    116120}}}
    117121
    118 JRF: When checking that we could use the console commands on ipp113, it has some troublesome status:
     122
     123JRF: When checking that we could use the console commands on ipp113, it has some troublesome status. MEH will report this to Hayden.
    119124{{{
    120125Outlet                Name                Status          Post-on Delay(s)
    121126i05_pdu0[15]          i48A_15_BAD         OFF(locked)     0.5
    122127}}}
     128N.B. To get out of the console, do: Shift + ~, .
     129That is press shift and tilde together (possibly twice), then release and press period.
    123130
    124131
    125132=== Thursday : 2019.02.07 ===
     133
     134JRF: ME, MEH, and TdB shall be going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area).
    126135
    127136=== Friday : 2019.02.08 ===