This is the page for the IPP Operational Configuration Management.

Operational Configuration Monitoring

Every day, at 06:30 HST, on ippc18, the ipp user runs the ~ipp/local/bin/configuration_monitoring.sh script. This script compares files which are used by the IPP in operation mode (I will call them actual files) with versions of those files (the expected files). This daily task is performed using crontab.

The expected files are controlled through SVN. The repositories location is on /data/ippc18.0/home/operational_repository (thus on ippc18). Synchronization of this directory is performed every day (the rsync completion/failure status is shown in the daily report).

The current operational configuration repository is in the current project. Tagged versions (none for the moment) should be placed as subproject of the tags project.

Any member of the users group (on the Maui cluster) can export/checkout the current operational configuration:

svn export file:///data/ippc18.0/home/operational_repository/current

or

svn checkout file:///data/ippc18.0/home/operational_repository/current

The files that are checked (and the way to fix any check failure) are detailed in the following paragraphs.

How to fix failures

PanTasks Servers Configuration Files

The <pantasks_server>s which are managed in configuration are:

addstar
cleanup
detrend
distribution
pstamp
publishing
registration
replication
stack
stdscience
summitcopy
update

Each <pantasks_server> has its own directory under pantasks. Two files are managed in each directory:

input
ptolemy.rc

If you are responsible for a change in an input or ptolemy.rc file for some <pantasks_server> (if you are not the best is likely to ask ps-ipp-ops mailing-list). First go into the <pantasks_server> directory:

cd ~ipp/<pantasks_server>

Update and then inspect its contents (according to SVN):

svn update .
svn status . -q

The input or the ptolemy.rc (or both) should be displayed with a 'M'.

Modified input file

svn diff input

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" input

Modified ptolemy.rc file

svn diff ptolemy.rc

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" ptolemy.rc

ippconfig Directory Files

Four files are managed in configuration in the operational ippconfig directory:

ippmonitor.config
ipprc.config
nebulous.config
site.config

If you are responsible for a change in the operational ippconfig directory (if you are not, the best is likely to ask ps-ipp-ops mailing-list). First go into the operational ippconfig directory:

cd ~ipp/ippconfig

Update and then inspect its contents (according to SVN):

svn update .
svn status . -q

One of the files should be shown with an 'M'.

Modified ippmonitor.config file

svn diff ippmonitor.config

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" ippmonitor.config

Modified ipprc.config file

svn diff ipprc.config

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" ipprc.config

Modified nebulous.config file

svn diff nebulous.config

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" nebulous.config

Modified site.config file

svn diff site.config

If you are satisfied with the changes, as YOURSELF:

svn commit -m "`An informative message`" site.config

Databases Configuration Files

The databases configuration files which are checked are the MySql configuration files on the following hosts:

ippdb00: The master for Nebulous
ippdb01: The master for gpc1, isp, ippadmin,czardb
ippdb02: The replication slave of ippdb00 (for Nebulous)
ippdb03: The replication slave of ippdb01 (for gpc1, isp, ippadmin)
ippc17: The master for ippRequestServer
ippc19: The replication slave of ippc19 (for ippRequestServer)

Each expected my.cnf file (located in the current/<host>) is diff'd against the actual version /etc/mysql/my.cnf on the <host> (The file on the <host> is locally copied using scp and the local copy is diff'd).

If you are responsible for a check failure (that is, if you modified the /etc/mysql/my.cnf file on some <host>), you have to log as YOURSELF on the Maui cluster and do:

mkdir ~/tmp/ipp_ocm                                                       # Create a working directory
cd ~/tmp/ipp_ocm                                                          # Go into that working directory
svn checkout file:///data/ippc18.0/home/operational_repository/current    # Get the current repository contents
cd current/databases/<host>                                               # Change to the "right" host directory
scp <host>:/etc/mysql/my.cnf .                                            # Copy the MySql configuration file to your local working copy
svn commit -m "`Some informal message`" .                                 # Commit the changes

Databases Backup Files

The Databases Backup related Files are used to perform and manage the various MySql dumps. They are located in ~ipp/mysql-dump:

README              # The documentation
gpc1Import.py       # Python script dedicated to gpc1_0/gpc1_1 imports
gpc1_dump.sh        # Script for gpc1, isp, ippadmin, czardb replication slave dump
gpc1_install.sh     # Script for gpc1, isp, ippadmin, czardb databases dump copy on ipp001
neb_copy.sh         # Script for Nebulous replication slave copy on ipp001
neb_dump.sh         # Script for Nebulous replication slave dump
ops_dump.csh        # Old script?
password.sh         # File  containing passwords for db access

To fix a check failure, files modifications need to be committed in the current/databases/backup directory. The procedure is similar to the one shown in ​related to the Databases Configuration Files.

Nebulous Apache Configuration Files

These files are the Apache configuration files that were modified or added for Nebulous.

They are present on 10 hosts ippc01, ippc02, ippc03, ippc04, ippc05, ippc06, ippc07, ippc08, ippc09, ippc10 and two files are managed in configuration

httpd.conf
modules.d/apache2-mod_perl-startup.pl

Similarly to ​Databases Configuration Files, the files are copied from the <host> and compared to the expected version located in current/nebulous/<host>. Changes should be committed in the same way as described in that paragraph.

Other Files

Other files are files that do not fit into other categories.

• The operational version of check_system.sh is located in ~ipp. Its expected version is located in current/utilities.
• The operational version of configuration_monitoring.sh is located in ~ipp/local/bin. Its expected version is located in current/utilities.

Changes to one of those files need to be committed to fix any check failure.

RSync Failures

The SVN repository is rsync'ed every day at 06:30.

In case of failure look at the log file and if you can't figure out what happened, ask someone who knows about rsync.

Requests

Software Improvements/Bugs

Files / Directories to be added to the OCM

Please be precise and exhaustive (name of the file, location)

Software Changes

beta version

• rsync to a directory on the manoa cluster; something under /data/ipp001.0/ipp: DONE (but not on the Manoa cluster since there are operational passwords in the repository)
• E-mail title according to global status: DONE
• mysql configuration files: ippdb00, ippdb01, ippdb02, ippdb03, ipp001, ippc17, ippc19: DONE
• apache/nebulous configuration files: ippc01-ippc09, ippc10: DONE