Changes between Version 1 and Version 2 of FilesMonitoring

Timestamp:

Dec 11, 2012, 11:45:24 AM (14 years ago)

Author:

Serge CHASTEL

Comment:

--

FilesMonitoring

@@ -11 +11 @@
  * The source code is in {{{~schastel/dev/FilesMonitoring}}}. In the rest of this documentation, all paths are relative to it. TODO: SVN it
  * On each storage node, every hour, the {{{NodeRelated/refresh_table.py}}} script is started (from the crontab file defined in {{{NodeRelated/deployment/scripts/crontab.<node name>}}}). To prevent cluster/nfs overwhelming, the scripts are not run at the same time. The crontab files as well as the deployment script can be generated from {{{NodeRelated/deployment/generate_crontabs_and_deployment_script.py}}}. Run it and follow its (last) instruction to deploy. Note that error messages are currently sent to {{{schastel@ifa.hawaii.edu}}}.
+ * Every 4 hours (see schastel crontab on ippc11), the {{{Summary/snapshot.py}}} script is run and takes snapshots, i.e. statistics, of each storage node database.
+
+= {{{NodeRelated/refresh_table.py}}} Details =
+
+== User Doc ==
+ * This script runs on each storage node. Each storage node should run a number of instances on this script equal to the number of its storage partitions (i.e. 1 for the mhpcc nodes, 3 for the atrc and the stsci nodes).
+ * The script uses the local mysql server and its {{{DisksMonitoring}}} database
+ * '''If the script (resp. the mysql server) are overwhelming the host (nfs, I/O, swap...), FEEL FREE TO KILL (resp. STOP) THEM'''. If you stop the mysql server, think of restarting it. The script should be robust enough to deal with it in the worst case, it will send an e-mail to the MAILTO recipient defined in {{{NodeRelated/deployment/scripts/crontab.<node name>}}}.
+
+== Script Stages ==
+ * Constants are (should be) all defined in the Constants pseudo-class
+ * Some initializations are performed (which disk is the script supposed to analyze, logging, ...)
  * 
+
+== Deployment ==
+TODO: 
+
+= {{{NodeRelated/update_table.py}}} Details =
+ * The script depends on {{{ipp.lifetime_templates}}}
+
+= {{{~schastel/local/share/python/ipp/lifetime_templates.py}}} Details =
+
+See [wiki:Cluster_Storage_Notes]
+
+= {{{Summary/refresh_table.py}}} Details =
+
+
+= Tools =
+
+== {{{deneb_locate.py}}}: Locate a file using its nebulous storage key ==
+ * The "nebulous storage key" is something like gpc1/sc_test_01/stack/MD08.V2/skycell.128/MD08.V2.skycell.128.stk.418540.ppStack.mdc
+ * Equivalent of the {{{neb-ls -p}}} command but uses the different {{{DisksMonitoring}}} databases;
+ * Is installed in {{{~ipp/local/bin}}} and can be executed as user {{{ipp}}};
+ * {{{-h}}} to get help.
+ 
+== {{{}}} TODO Statistics ==
+ *