IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 5 and Version 6 of FilesMonitoring


Ignore:
Timestamp:
Dec 11, 2012, 12:44:59 PM (14 years ago)
Author:
Serge CHASTEL
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • FilesMonitoring

    v5 v6  
    5656TODO: We don't like duplicate code, do we?
    5757= {{{~schastel/local/share/python/ipp/lifetime_templates.py}}} Details =
    58 
    5958See [wiki:Cluster_Storage_Notes]
    6059
     60This is where the file classifier and the classification definition are both defined. Since python allows easy dynamic class loading there is not need to have distinct definitions.
     61
     62Each file name is classified:
     63 * The stage is determined (chip, cam, ..., unknown): This is the stage where the file is supposed to have been built;
     64 * The role is determined (b1fits, stats, ..., unknown): This is the role of the file;
     65 * The lifetime of the file is obtained from both the stage and the role: raw (keep the file till the end of times and possibly after), permanent (a product to be kept), ephemeral (product that will be cleaned up one day), unknown (it's unknown as soon as one is unknown)
     66
     67To adding a new stage, a new role:
     68 * The {{{initialize_IppTemplates}}} function has to be updated;
     69 * Add a stage definition in the first lines of the  {{{initialize_IppTemplates}}} function. A stage definition is made of a unique name (a Python str) and a Python list of regular expressions (a regular expression is a string as well). You need to define the lifetime of the role you want to the next lines. For instance, when the detrend stage was added, I added:
     70{{{
     71 Stage.add_stage(Stage( 'detrend', [ '^.*\.det.*$', ] ))
     72}}}
     73and then:
     74{{{
     75    Lifetime.add_lifetime( Lifetime('detrend', 'b1fits', 'permanent') )
     76...
     77    Lifetime.add_lifetime( Lifetime('detrend', 'b1jpg', 'permanent') )
     78}}}
     79Note: I try to keep it clean and easily maintanable by grouping the lifetime definitions under the role associated to it.
     80 * It is perfectly acceptable to have a stage with no associated lifetime (e.g. skycal and staticsky)
    6181= {{{Summary/refresh_table.py}}} Details =
    6282