IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 29438


Ignore:
Timestamp:
Oct 15, 2010, 2:51:13 PM (16 years ago)
Author:
Serge CHASTEL
Message:

Test report now in xml

Location:
branches/sc_branches/psps_testing
Files:
1 added
10 edited

Legend:

Unmodified
Added
Removed

  • branches/sc_branches/psps_testing/README

    r29117 r29438  
    1 1. Testing PSPS ingestion
    2 =========================
    3 1.0 Currently:
    4    Roy's 10 inputs are half(well... less than half)-tested through build.py
     1This is the PSPS test framework. Currently, it allows:
     2- Testing itself;
     3- Testing if data ingestion is ok. See part 2;
     4- That's it.
    55
    6 1.1 Testing a valid input
    7    (not yet implemented but will be)
    8    psps_test.py <input.tar.gz>
    9 
    10 2. Testing the framework
    11 ========================
    12 2.1. Running the unit tests:
     61. Testing the test framework
     7=============================
     81.1. Running the unit tests:
    139./build.sh unittest
    1410
    15 2.2. Coverage
     111.2. Coverage
    1612./build.sh coverage
    1713
     141.3. Documentation
     15Generate it with
     16./build.sh documentation
     17
     182. Testing PSPS data ingestion
     19==============================
     202.1 Testing a valid input
     21      (not yet implemented but will be)
     22    psps_test.py <input.tar.gz> <report.xml>
     23    - <input.tar.gz>: IPP batch (output of ipp2psps)
     24    - <report.xml>: Test report
     25      (can be converted to html and/or pdf using TODO)
     26
     27(currently, edit psps_test.py and replace input file name)
     28

  • branches/sc_branches/psps_testing/build.sh

    r29235 r29438  
    77
    88PYTHON=/usr/bin/python
     9UNITTEST_LOG=unittest.log
    910COVERAGE=/usr/bin/python-coverage
     11DOCUMENTATION=/usr/bin/epydoc
     12DOC_FOLDER=documentation
     13# Check epydoc available formats
     14DOC_FORMAT=html
    1015
    1116FILES="testers/__init__.py \
    … …  
    1924  utilities/file_manipulation.py \
    2025  utilities/psps_logger.py \
     26  utilities/abstract_test_report.py \
    2127  utilities/test_report.py \
    2228  utilities/test_product.py \
    … …  
    2834case "$1" in
    2935    unittest)
     36        something_wrong=0
     37        /bin/rm -f $UNITTEST_LOG
    3038        for file in $FILES; do
    31             echo "Unit test for $file ($PYTHON -m doctest $file)"
    32             $PYTHON -m doctest $file
     39            echo -n "Unit test for $file ($PYTHON -m doctest $file): "
     40            $PYTHON -m doctest $file >> $UNITTEST_LOG 2>> $UNITTEST_LOG
     41            status=$?
     42            if [ "$status" -eq "0" ]; then
     43                echo "OK"
     44            else
     45                echo "KO"
     46                (( something_wrong= $something_wrong + 1 ))
     47            fi
    3348        done
     49        if [ "$something_wrong" -eq "0" ]; then
     50            echo "All tests pass"
     51        else
     52            echo "!!! $something_wrong tests failed"
     53            echo "!!! Look at $UNITTEST_LOG for details"
     54        fi
    3455        ;;
    3556    coverage)
    … …  
    4061        echo "### Coverage results ###"
    4162        $COVERAGE -r -o /usr,$HOME/local -m
     63        ;;
     64    doc|documentation)
     65        $DOCUMENTATION -o $DOC_FOLDER --$DOC_FORMAT `find . -name \*.py | grep -v weirdos | grep -v html | grep -v create_simple_fits` -v
     66        echo "Documentation is at:"
     67        echo "  file://`pwd`/documentation/index.html"
    4268        ;;
    4369    all)
    … …  
    5177        ;;
    5278    *)
    53         echo "Usage: $0 [ all | unittest | coverage]"
     79        echo "Usage: $0 [ all | unittest | coverage | [doc|documentation]]"
    5480        exit 1
    5581        ;;

  • branches/sc_branches/psps_testing/psi/psi_inquisitor.py

    r29235 r29438  
    22from suds.xsd.doctor import Import, ImportDoctor
    33from suds import sudsobject
     4from suds import WebFault
    45from utilities.util import psi_convert
    56import unicodedata
    67from utilities.psps_logger import PsPsLogger
    78from utilities.test_report import TestReport
     9
     10from suds import WebFault
     11class PsiException(WebFault):
     12    """
     13    >>> fault = WebFault('fault', 'document')
     14    >>> raise PsiException(fault)
     15    Traceback (most recent call last):
     16    ...
     17    PsiException
     18    """
     19    def __init__(self, webFault):
     20        WebFault.__init__(self, webFault.fault, webFault.document)
    821
    922class PsiInquisitor:
    … …  
    3144        TestReport.info('PsiInquisitor created from %s' % configuration_file)
    3245        TestReport.info(self.configuration.str())
     46        Client.log = PsPsLogger
    3347        self.authClient = Client(self.configuration.authUrl)
    3448        self.sessionID = self.authClient.service.login(self.configuration.userID,
    … …  
    104118        """
    105119        PsPsLogger.debug('Querying PSI: [%s]' % sql_query)
    106         results = self.jobsClient.service.executeQuickJob(self.sessionID,
    107                                                           self.configuration.schemaGroup,
    108                                                           sql_query,
    109                                                           self.configuration.context)
     120        try:
     121            results = self.jobsClient.service.executeQuickJob(self.sessionID,
     122                                                              self.configuration.schemaGroup,
     123                                                              sql_query,
     124                                                              self.configuration.context)
     125        except WebFault, e:
     126            raise PsiException(e)
    110127        lines = results.split('\n')
    111128        first_line_not_seen = True

  • branches/sc_branches/psps_testing/psi/web01_configuration.py

    r29227 r29438  
    2626          JobsUrl: http://web01.psps.ifa.hawaii.edu/DFetch/WSDL/JobsService.php.wsdl
    2727        """
    28         return """Configuration
     28        return """PsiConfiguration
    2929  Server: %s
    3030  User: %s

  • branches/sc_branches/psps_testing/testers/batch_file.py

    r29235 r29438  
    2121    The purpose of this class is to test the common features of batch
    2222    files. Namely, they are:
    23     - the form of the file name
    24     - the contents of the file (a gzipped tar archive file) and the form
    25       of the files contained in the archive
     23      - the form of the file name
     24      - the contents of the file (a gzipped tar archive file) and the form
     25        of the files contained in the archive
    2626
    2727    >>> print BatchFileTester("data/psut/ok/B00000010.tar.gz").test().success
    28     False
    29     >>> # Will be True one day...
     28    True
    3029    """
    3130    def __init__(self, filename):
    … …  
    5958        """
    6059        PsPsLogger.debug('Starting BatchFileTester object test')
    61         TestReport.title('Batch file tests')
     60        TestReport.addSection('Batch file tests')
    6261        # Set up global test product
    6362        product = TestProduct()
    … …  
    166165        Derived Requirement 3.3.1: The maximum size of a batch file
    167166        shall not exceed 100 gigabytes (TBR).
    168 
    169         -> Test 1: The file is a valid tar gzipped file
    170         -> Test 2: The file uncompresses in the <basename> directory
    171                 where <basename> is the file base name
    172         -> Test 3: The tar gzipped file contains exactly one XML file
    173               (named BatchManifest.xml) and exactly one FITS file.
     167       
     168        Test 1: The file is a valid tar gzipped file
     169
     170        Test 2: The file uncompresses in the <basename> directory
     171        where <basename> is the file base name
     172
     173        Test 3: The tar gzipped file contains exactly one XML file
     174        (named BatchManifest.xml) and exactly one FITS file.
    174175
    175176        >>> batchFileTester = BatchFileTester('data/psut/ok/B00000010.tar.gz')

  • branches/sc_branches/psps_testing/testers/batch_manifest_file.py

    r29235 r29438  
    7171        00000010.FITS
    7272        """
    73         TestReport.title('Batch manifest file tests')
     73        TestReport.addSection('Batch manifest file tests')
    7474        # Check batch file existence and correct naming in archive
    7575        if not self._test_batch_file_exists().success:

  • branches/sc_branches/psps_testing/testers/fits/p2.py

    r29228 r29438  
    4949        True
    5050        """
    51         PsPsLogger.info('Testing P2FitsTester object')
    52         # 1. There is no image for the PrimaryHDU (at 0)
    53         PsPsLogger.debug('  Testing P2FitsTester object: primary')
    54         product = self.test_primary()
    55         if not product.success:
    56             return product
    57         # 2. Look at the FrameMeta information
    58         PsPsLogger.debug('  Testing P2FitsTester object: frameMeta')
    59         product = self.test_frame_meta(be_tolerant_for_types)
    60         # Fail if there is any failure and we are type-strict
    61         if not product.success and not be_tolerant_for_types:
    62             return product
    63         # 3. There should be 'product.nOTA' sequences of [ImageMeta,
    64         # Detection, SkinnyObject, ObjectCalColor] frames
    65         frameID = product.frameID
    66         nOTA = product.nOTA
    67         if len(self.fits) != 4*nOTA + 2:
    68             TestReport.ko('Invalid number of OTA: from FITS file: %d / from FrameMeta %d'
    69                           % (len(self.fits)-2/4, nOTA),
     51        try:
     52            PsPsLogger.info('Testing P2FitsTester object')
     53            # 1. There is no image for the PrimaryHDU (at 0)
     54            PsPsLogger.debug('  Testing P2FitsTester object: primary')
     55            product = self.test_primary()
     56            if not product.success:
     57                return product
     58            # 2. Look at the FrameMeta information
     59            PsPsLogger.debug('  Testing P2FitsTester object: frameMeta')
     60            product = self.test_frame_meta(be_tolerant_for_types)
     61            # Fail if there is any failure and we are type-strict
     62            if not product.success and not be_tolerant_for_types:
     63                return product
     64            # 3. There should be 'product.nOTA' sequences of [ImageMeta,
     65            # Detection, SkinnyObject, ObjectCalColor] frames
     66            frameID = product.frameID
     67            nOTA = product.nOTA
     68            if len(self.fits) != 4*nOTA + 2:
     69                TestReport.ko('Invalid number of OTA: from FITS file: %d / from FrameMeta %d'
     70                              % (len(self.fits)-2/4, nOTA),
     71                              {'requirement': 'TBD'})
     72                return TestProduct()
     73            for ota_index in range(nOTA):
     74                PsPsLogger.debug('  Testing P2FitsTester object: OTA (%d/%d)'
     75                                 % ((ota_index+1), nOTA))
     76                subproduct = self.test_detections_frames(frameID, ota_index,
     77                                                         be_tolerant_for_types)
     78                if subproduct.success:
     79                    TestReport.ok(requirement)
     80                product.success = product.success and subproduct.success
     81            return product
     82        except KeyError, e:
     83            PsPsLogger.error('  Exception caught!')
     84            TestReport.ko('Unexpected exception',
    7085                          {'requirement': 'TBD'})
     86            TestReport.info(e)
    7187            return TestProduct()
    72         for ota_index in range(nOTA):
    73             PsPsLogger.debug('  Testing P2FitsTester object: OTA (%d/%d)'
    74                                % ((ota_index+1), nOTA))
    75             subproduct = self.test_detections_frames(frameID, ota_index,
    76                                                      be_tolerant_for_types)
    77             if subproduct.success:
    78                 TestReport.ok(requirement)
    79             product.success = product.success and subproduct.success
    80         return product
     88           
    8189
    8290    #####################################################
    … …  
    9199        fits_index = 4*ota_index+2
    92100        product = TestProduct()
    93         TestReport.title('Data frames tests')
     101        TestReport.addSection('Data frames tests')
    94102        if self.fits[fits_index].name != 'IMAGEMETA':
    95103            TestReport.ko(requirement,
    … …  
    115123        query = 'SELECT * FROM ImageMeta WHERE imageID = %s' % str(imageID)
    116124        answer = self.psi_inquisitor.query(query)
    117         TestReport.title('ImageMeta values tests (frame %d - %s)'
    118                          % (ota_index, query))
     125        TestReport.addSection('ImageMeta values tests (frame %d - %s)'
     126                              % (ota_index, query))
    119127        subproduct = match(self.fits[fits_index].data, answer, requirement,
    120128                           tolerance = Configuration.TOLERANCE,
    … …  
    158166        """
    159167        Tests the contents of the FrameMeta table.
    160         - 48 columns x 1 row
    161         - This table contains values that are useful, namely: frameID,
    162           nOTA (number of OTA).
    163         - The data can be found in PSI using the query:
     168          - 48 columns x 1 row
     169          - This table contains values that are useful, namely: frameID,
     170            nOTA (number of OTA).
     171          - The data can be found in PSI using the query:
    164172                   SELECT * FROM FrameMeta WHERE frameID = <frameID>;
    165173        """
    166174        requirement = 'PSDC-940-006-01, 3.6.6.8: '
    167175        product = TestProduct()
    168         TestReport.title('MetaFrame tests')
     176        TestReport.addSection('MetaFrame tests')
    169177        if self.fits[1].header['NAXIS'] != 2:
    170178            TestReport.ko(requirement, 'NAXIS != 2')
    … …  
    196204        Check that the primary header i.e. fits[0] has no dimension.
    197205        """
    198         TestReport.title('Primary frame tests')
     206        TestReport.addSection('Primary frame tests')
    199207        requirement = 'PSDC-940-006-01, 3.6.3, 1'
    200208        product = TestProduct()

  • branches/sc_branches/psps_testing/utilities/psps_logger.py

    r29235 r29438  
    123123        import doctest
    124124        doctest.testmod()
     125        PsPsLogger.set_fileout('/dev/null')
    125126        PsPsLogger.test()
    126127        sys.exit(0)

  • branches/sc_branches/psps_testing/utilities/test_report.py

    r29235 r29438  
    22import datetime
    33from utilities.configuration import Configuration
     4from utilities.abstract_test_report import AbstractTestReport
    45
    5 class TestReportClass(logging.Logger):
    6     datefmt = '%H:%M:%S'
    7     fmt = '[%(asctime)8s] | %(requirement)65s | %(message)s'
    8     class __impl(logging.Logger):
     6class TestReportClass(AbstractTestReport):
     7    """
     8    An implementation of AbstractTestReport for PSPS testing test
     9    reports.
     10    """
     11    class __impl(AbstractTestReport):
    912        """ Implementation of the singleton interface """
    1013        def __init__(self):
    11             """Sets logging up"""
    12             logging.Logger.__init__(self, __name__)
    13             logging.basicConfig(level=logging.INFO,
    14                                 format=TestReportClass.fmt)
    15             handler = logging.StreamHandler(open('log/test_report.txt', 'w'))
    16             handler.setFormatter(logging.Formatter(TestReportClass.fmt,
    17                                                    TestReportClass.datefmt))
    18             self.addHandler(handler)
     14            """Sets up AbstractTestReport"""
     15            AbstractTestReport.__init__(self)
    1916        def id(self):
    2017            """ Test method, return singleton id """
    … …  
    2320    __instance = None
    2421
    25     def info(self, message):
    26         for line in message.split('\n'):
    27             logging.Logger.info(self,
    28                                 line,
    29                                 extra = {'requirement': 'INFO'})
     22    def info(self, message, extra=None):
     23        self.addTestResult(AbstractTestReport.DEFAULT,
     24                           'INFO',
     25                           '',
     26                           message)
    3027
    3128    def __init__(self):
    … …  
    4441        # Store instance reference as the only member in the handle
    4542        self.__dict__['_TestReportClass__instance'] = TestReportClass.__instance
    46         logging.Logger.info(self,
    47                             'Test performed on ' + str(datetime.date.today()),
    48                             extra = {'requirement': 'This column usually shows the "Requirement ID"'})
     43        self.info('Test performed on ' + str(datetime.date.today()),
     44                  extra = {'requirement':
     45                           'This column usually shows the "Requirement ID"'})
    4946        self.info(str(Configuration()))
    5047
    … …  
    5754        return setattr(self.__instance, attr, value)
    5855
    59     # def test_result(self, requirement, message):
    60     #     self.info(message,
    61     #               extra = {'requirement': requirement})
    62     def title(self, title):
    63         logging.Logger.info(self,
    64                             title,
    65                             extra = {'requirement': '--------------------'})
    66     def ok(self, requirement):
    67          logging.Logger.info(self,
    68                              'OK',
    69                              extra = {'requirement': requirement})
    70     def ko(self, requirement, cause):
    71          logging.Logger.info(self,
    72                              'KO (%s)' % cause,
    73                              extra = {'requirement': requirement})
     56    def save(self, filename, prettyxml=True):
     57        f = open(filename, "w")
     58        if prettyxml:
     59            f.write(self.toprettyxml())
     60        else:
     61            f.write(self.toxml())
     62        f.close()
    7463
    7564TestReport = TestReportClass()
    7665
    77 if __name__ == '__main__':
     66if __name__ == '__main__': # pragma: no cover
    7867    import logging
    7968    import sys
    8069    current_argument_position = 1
    8170    while current_argument_position < len(sys.argv):
    82         if sys.argv[current_argument_position] == '-debug': # pragma: no cover
     71        if sys.argv[current_argument_position] == '-debug':
    8372            PsPsLogger.setLevel(logging.DEBUG)
    8473            PsPsLogger.set_stderr()
    85             current_argument_position += 1
    86         elif sys.argv[current_argument_position] == 'test':
    87             print 'Running unittest for TestReport class'
     74        elif sys.argv[current_argument_position] == 'unittest':
    8875            import doctest
    8976            doctest.testmod()
    9077            sys.exit(0)
     78        current_argument_position += 1
     79    print('  Usage: %s [--debug] unittest' % sys.argv[0])

  • branches/sc_branches/psps_testing/utilities/util.py

    r29235 r29438  
    125125                if complain:
    126126                    TestReport.ko(requirement,
    127                                   'Classes of FITS value (%s/%s) and PSI value (%s/%s) are different for key \'%s\''
    128                                   % (fits_value.__class__.__name__,
    129                                      fits_value,
    130                                      value.__class__.__name__,
    131                                      value,
    132                                      str(key)))
     127                                    'Classes of FITS value (%s/%s) and PSI value (%s/%s) are different for key \'%s\''
     128                                    % (fits_value.__class__.__name__,
     129                                       fits_value,
     130                                       value.__class__.__name__,
     131                                       value,
     132                                       str(key)))
    133133                failures_count += 1
    134134                complain = complain and __still_complain(max_failures, failures_count)
    … …  
    142142                if complain:
    143143                    TestReport.ko(requirement,
    144                                   'Different values for key %s: PSI = [%s] (%s) / FITS = [%s] (%s) / FITS index = %d' %
    145                                   (key,
    146                                    psi_value,
    147                                    psi_value.__class__.__name__,
    148                                    fits_data.field(key)[fits_index],
    149                                    fits_data.field(key)[fits_index].__class__.__name__,
    150                                    fits_index))
     144                                    'Different values for key %s: PSI = [%s] (%s) / FITS = [%s] (%s) / FITS index = %d' %
     145                                    (key,
     146                                     psi_value,
     147                                     psi_value.__class__.__name__,
     148                                     fits_data.field(key)[fits_index],
     149                                     fits_data.field(key)[fits_index].__class__.__name__,
     150                                     fits_index))
    151151                failures_count += 1
    152152                complain = complain and __still_complain(max_failures, failures_count)
    … …  
    158158            # Write the summary of all errors
    159159            TestReport.ko(requirement,
    160                           'Total of %d comparison errors for match (comparison over %d rows of %d columns)'
    161                           % (failures_count,
    162                              len(psi_answer['items']),
    163                              len(items.items())))
     160                            'Total of %d comparison errors for match (comparison over %d rows of %d columns)'
     161                            % (failures_count,
     162                               len(psi_answer['items']),
     163                               len(items.items())))
    164164    return product
    165165
Note: See TracChangeset for help on using the changeset viewer.