Changes between Version 34 and Version 35 of Cluster_Storage_Notes

Timestamp:

Oct 19, 2011, 4:50:53 PM (15 years ago)

Author:

watersc1

Comment:

--

Cluster_Storage_Notes

@@ -3 +3 @@
 == Automatic Storage Monitoring ==
 
-Cron jobs are set up on all nebulous storage nodes to scan the nebulous disks and construct statistics about their contents.  These jobs are striped across the cluster throughout the week, in an attempt to prevent all hosts running the disk scan simultaneously (which would presumably have an effect on processing throughput).  The usage statistics are placed in nebulous in files named `neb://ipp_diskspace/YYYY-MM-DD/ippXXX.Z.neb_usage.dat`, with the date string matching the date of the Sunday of the week the scan was performed.  The script parses each filename in the nebulous directory, and attempts to classify the stage it comes from as well as the type of product from that stage.  The total count of these files, and the total usage (in bytes) of these files.  These totals are what is stored in the statistics files.  An excerpt of one of these files is listed below, showing the CHIP stage file statistics from one host.
+Cron jobs are set up on all nebulous storage nodes to scan the nebulous disks and construct statistics about their contents.  These jobs (calling `ipp/trunk/tools/diskspace/nebulous_data.pl`) are striped across the cluster throughout the week, in an attempt to prevent all hosts running the disk scan simultaneously (which would presumably have an effect on processing throughput).  The usage statistics are placed in nebulous in files named `neb://ipp_diskspace/YYYY-MM-DD/ippXXX.Z.neb_usage.dat`, with the date string matching the date of the Sunday of the week the scan was performed.  The script parses each filename in the nebulous directory, and attempts to classify the stage it comes from as well as the type of product from that stage.  The total count of these files, and the total usage (in bytes) of these files.  These totals are what is stored in the statistics files.  An excerpt of one of these files is listed below, showing the CHIP stage file statistics from one host.
 {{{
 CHIP B1FITS 774142 320650755840
@@ -26 +26 @@
 }}}
 
-At the end of the week, the gpc1 database is polled to identify the count of states for each stage, and the results stored in `neb://ipp_diskspace/YYYY-MM-DD/run_im_counts.dat`.  This gives the stage, the data_state, and then the count of the number of components with that data_state:
+At the end of the week, the gpc1 database is polled by `ipp/trunk/tools/diskspace/run_counter.pl` to identify the count of states for each stage, and the results stored in `neb://ipp_diskspace/YYYY-MM-DD/run_im_counts.dat`.  This gives the stage, the data_state, and then the count of the number of components with that data_state:
 {{{
 CHIP cleaned 16945531
@@ -36 +36 @@
 CHIP update 3304
 }}}
-At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) that gives the usage in TB of each stage/product combination:
+At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/merge_counts_and_sizes.pl` that gives the usage in TB of each stage/product combination:
 {{{
 CHIP_B1FITS 4876.40501260757
@@ -61 +61 @@
 }}}
 
-A largely untested final summary should be generated that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  Running this script on the count file from 2011-10-09 yields (in part):
+A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/generate_mappings.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  Running this script on the count file from 2011-10-09 yields (in part):
 {{{
 CHIP     cleaned            PRODUCT        31179.7770     0.001840  16945531