Changes between Version 2 and Version 3 of nebulous-performance-analysis

Timestamp:

Feb 17, 2011, 12:09:24 PM (15 years ago)

Author:

Serge CHASTEL

Comment:

--

nebulous-performance-analysis

@@ -5 +5 @@
 == Framework and Tools ==
 
+The tools for analysis come from https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/trunk/tools/log_parsing.
+Namely, the script called nebulous_activity.py is run (its argument being the name of the directory where the Apache logs are stored. The script logs what it is doing on /dev/stderr and its results on /dev/stdout. Under bash, the best is therefore to run it this way:
+{{{
+   python ./nebulous_activity.py > results 2> execution.log
+}}}
+so that the execution can be easily monitored.
+
+The results are not sorted (by date). This can be done by: sort results > results.sorted. The results output format is a SSV (Space Separated Values) file organized in lines (end of line is \n). A result line starts by a timestamp (e.g. 2011-02-13T21:40:00) and numbers detailing the Apache/Nebulous activity during the 10 minutes following the timestamp (that is what happened between 2011-02-13T21:40:00 and 2011-02-13T21:49:59+(1-epsilon)). The first number is the total of Nebulous related messages in the access_log; the second one is the number of entries in the error_log; the third is the number of Nebulous create requests messages, the fourth is the number of find_instance requests; the fifth is the number of delete requests; the sixth is the number of fail_stat requests; the seventh of stat requests; the eighth the Nebulous unknown request (hence the first number is the sum of columns 3 to 8); the last/ninth column is the count of non-Nebulous requests.
+
+[<timestamp>_<Nebulous messages> ]
+
+Note: parsing log files can last LONG (Rule of the thumb 1 hour for 1 million lines for ippc00 (RAM 32G/IntelXeon 2.4Ghz, cache 6M, no RAID?). If you find the law, write it down)
+
+Note: I have to mention the existence of [http://code.google.com/p/apachelog/ apachelog] which is a suite of Python utilities to parse Apache log files. I did not used it because we are not interested in details of Apache (e.g. hosts where the requests come from) but in Nebulous activity. If one day, we need more details, think about using it.
+
 == Input Data: Case Study ==
 
+I took data from the current ippdb00:/var/log/apache2/ directory, where current means 2011-02-15T16:18:59). The directory contains access and error logs for the period 2011-01-31T16:20:04 to current date.
+
+
+
 == Anything else? ==