This is the page for the IPP Operational Configuration Management. [[PageOutline]] = Operational Configuration Monitoring = == How it works == Every day, at 06:30 HST, on {{{ippc18}}}, the {{{ipp}}} user runs the {{{~ipp/local/bin/configuration_monitoring.sh}}} script. This script compares files which are used by the IPP in operation mode (I will call them ''actual'' files) with versions of those files (the ''expected'' files). This daily task is performed using crontab. The expected files are controlled through SVN. The repositories location is on {{{/data/ippc18.0/home/operational_repository}}} (thus on ippc18). Synchronization of this directory is performed every day (the rsync completion/failure status is shown in the daily report). The current operational configuration repository is in the {{{current}}} project. Tagged versions (none for the moment) should be placed as subproject of the {{{tags}}} project. Any member of the users group (on the Maui cluster) can export/checkout the current operational configuration: {{{ svn export file:///data/ippc18.0/home/operational_repository/current }}} or {{{ svn checkout file:///data/ippc18.0/home/operational_repository/current }}} The files that are checked (and the way to fix any check failure) are detailed in the following paragraphs. == Product == The script sends an e-mail when it is finished. = Checks / How to Fix Checks Failures = == PanTasks Servers Configuration Files == The {{{}}}s which are managed in configuration are: {{{ addstar cleanup detrend distribution pstamp publishing registration replication stack stdscience summitcopy update }}} Each {{{}}} has its own directory under {{{pantasks}}}. Two files are managed in each directory: {{{ input ptolemy.rc }}} === Fixing check failures === If you are responsible for a change in an input or ptolemy.rc file for some (if you are not the best is likely to ask ps-ipp-ops mailing-list). First go into the directory: {{{ cd ~ipp/ }}} Update and then inspect its contents (according to SVN): {{{ svn update . svn status . -q }}} The {{{input}}} or the {{{ptolemy.rc}}} (or both) should be displayed with a 'M'. === Modified {{{input}}} file === {{{ svn diff input }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" input }}} === Modified {{{ptolemy.rc}}} file === {{{ svn diff ptolemy.rc }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" ptolemy.rc }}} == {{{ippconfig}}} Directory Files == Four files are managed in configuration in the operational {{{ippconfig}}} directory: {{{ ippmonitor.config ipprc.config nebulous.config site.config }}} If you are responsible for a change in the operational {{{ippconfig}}} directory (if you are not, the best is likely to ask ps-ipp-ops mailing-list). First go into the operational {{{ippconfig}}} directory: {{{ cd ~ipp/ippconfig }}} Update and then inspect its contents (according to SVN): {{{ svn update . svn status . -q }}} One of the files should be shown with an 'M'. === Modified {{{ippmonitor.config}}} file === {{{ svn diff ippmonitor.config }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" ippmonitor.config }}} === Modified {{{ipprc.config}}} file === {{{ svn diff ipprc.config }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" ipprc.config }}} === Modified {{{nebulous.config}}} file === {{{ svn diff nebulous.config }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" nebulous.config }}} === Modified {{{site.config}}} file === {{{ svn diff site.config }}} If you are satisfied with the changes, as YOURSELF: {{{ svn commit -m "`An informative message`" site.config }}} == Databases Configuration Files == The databases configuration files which are checked are the MySql configuration files on the following hosts: {{{ ippdb00: The master for Nebulous ippdb01: The master for gpc1, isp, ippadmin,czardb ippdb02: The replication slave of ippdb00 (for Nebulous) ippdb03: The replication slave of ippdb01 (for gpc1, isp, ippadmin) ippc17: The master for ippRequestServer ippc19: The replication slave of ippc19 (for ippRequestServer) }}} Each expected my.cnf file (located in the {{{current/}}}) is diff'd against the actual version {{{/etc/mysql/my.cnf}}} on the {{{}}} (The file on the {{{}}} is locally copied using {{{scp}}} and the local copy is diff'd). If you are responsible for a check failure (that is, if you modified the {{{/etc/mysql/my.cnf}}} file on some {{{}}}), you have to log as YOURSELF on the Maui cluster and do: {{{ mkdir ~/tmp/ipp_ocm # Create a working directory cd ~/tmp/ipp_ocm # Go into that working directory svn checkout file:///data/ippc18.0/home/operational_repository/current # Get the current repository contents cd current/databases/ # Change to the "right" host directory scp :/etc/mysql/my.cnf . # Copy the MySql configuration file to your local working copy svn commit -m "`Some informal message`" . # Commit the changes }}} == Databases Backup Files == The Databases Backup related Files are used to perform and manage the various MySql dumps. They are located in {{{~ipp/mysql-dump}}}: {{{ README # The documentation gpc1Import.py # Python script dedicated to gpc1_0/gpc1_1 imports gpc1_dump.sh # Script for gpc1, isp, ippadmin, czardb replication slave dump gpc1_install.sh # Script for gpc1, isp, ippadmin, czardb databases dump copy on ipp001 neb_copy.sh # Script for Nebulous replication slave copy on ipp001 neb_dump.sh # Script for Nebulous replication slave dump ops_dump.csh # Old script? password.sh # File containing passwords for db access }}} To fix a check failure, files modifications need to be committed in the {{{current/databases/backup}}} directory. The procedure is similar to the one shown in [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/IPP_Operational_Configuration_Management#DatabasesConfigurationFiles related to the Databases Configuration Files]. == Nebulous Apache Configuration Files == These files are the Apache configuration files that were modified or added for Nebulous. They are present on 10 hosts {{{ippc01, ippc02, ippc03, ippc04, ippc05, ippc06, ippc07, ippc08, ippc09, ippc10}}} and two files are managed in configuration {{{ httpd.conf modules.d/apache2-mod_perl-startup.pl }}} Similarly to [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/IPP_Operational_Configuration_Management#DatabasesConfigurationFiles Databases Configuration Files], the files are copied from the {{{}}} and compared to the expected version located in {{{current/nebulous/}}}. Changes should be committed in the same way as described in that paragraph. == Other Files == Other files are files that do not fit into other categories. * The operational version of {{{check_system.sh}}} is located in {{{~ipp}}}. Its expected version is located in {{{current/utilities}}}. * The operational version of {{{configuration_monitoring.sh}}} is located in {{{~ipp/local/bin}}}. Its expected version is located in {{{current/utilities}}}. Changes to one of those files need to be committed to fix any check failure. == RSync Failures == The SVN repository is rsync'ed every day at 06:30. In case of failure look at the log file and if you can't figure out what happened, ask someone who knows about rsync. = Requests = == Software Improvements/Bugs == == Files / Directories to be added to the OCM == Please be precise and exhaustive (name of the file, location) = Software Changes = '''beta''' version * rsync to a directory on the manoa cluster; something under /data/ipp001.0/ipp: DONE (but not on the Manoa cluster since there are operational passwords in the repository) * E-mail title according to global status: DONE * mysql configuration files: ippdb00, ippdb01, ippdb02, ippdb03, ipp001, ippc17, ippc19: DONE * apache/nebulous configuration files: ippc01-ippc09, ippc10: DONE