Changes between Version 2 and Version 3 of ippToPspsOrr
- Timestamp:
- Feb 15, 2012, 5:34:23 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPspsOrr
v2 v3 1 2 [[PageOutline]] 1 3 2 4 = Some notes about ippToPsps for the PSPS Operational Readiness Review (ORR) = … … 16 18 === 2011 === 17 19 18 Stacks available for the first just prior to the Boston meeting in May.20 More loading and testing early in the year. 19 21 20 Many changes made to ippToPsps to improve speed in order to load nightly data as well as reprocessed: 21 - multi-client: multiple instances of ippToPsps can be run on the same machine, or multiple machines 22 - some processing (for likelihoods) now done by a MySQL stored procedure making a big speed increase 23 - areas of sky with high coverage (eg MD fields) are incredibly slow to access from DVO, so for these area we can pre-ingest DVO data into MySQL to speed up loading. 22 Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time. This prompted us to create a DV) to MySQL system meaning a whole DVO database could be ingested in MySQL for faster processing through ippToPsps. 24 23 25 With these improvements we stress-tested loading during November and December 2011. We loaded the new LAP data as it became available from the IPP and loaded half the sky (6-18 hrs RA) of old 3PI data. This included the galactic center with somes frames containing up to 4 million detections. 24 After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP). With the promise of a high throughput from IPP we needed to speed up ippToPsps, so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, in order to load batches in parallel. By using a secure critical section clients cannot ....TODO 26 25 27 Between Oct 27th and Dec 1st, i.e. 5 weeks, we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was 55,000.26 Another speed increase was obtained by implementing a stored-procedure in MySQL for the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps. 28 27 29 This was done simultaneously with the loading of LAP.28 During November and December we loaded half of all available old 3PI data from the IPP, while continuing to load new LAP data. 30 29 31 30 In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data. 32 31 33 For normal 3PI data in a quiet part of the sky we can easily load ~100 per hour, ~2400 exposures per day using multiple loading clients over up to 6 hosts. This system is flexible and new clients can be started and stopped with ease depending on requirements. 32 == Summary of major speed improvements to ippToPsps == 34 33 35 In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required. 34 In summary, the major developments to improve loading speed in 2011 have been: 35 36 * multi-client version 37 * stored procedure to calculate likelihoods 38 * ability to pre-ingest dense regions of sky from DVO into MySQL for 39 40 == Loading stress-test stats == 41 42 With the code improvements described above, we stress-tested loading through ippToPsps during November and December 2011 by loading a large chunk of old 3PI data from the IPP. This loading of old data was performed between Oct 27th and Dec 1st, i.e. 5 weeks in which time we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was roughly 55,000 frames. Some key points: 43 44 * this included the galactic center with somes frames containing up to 4 million detections 45 * this was done simultaneously with the loading of LAP frames and stacks 46 47 This stress-testing of ippToPsps showed that, for normal 3PI data in a quiet part of the sky, we can easily load '''~100 frames per hour''', or '''~2400 exposures per day''' using multiple loading clients on up to 6 hosts. In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required. 48 36 49 37 50 == Anatomy of a batch == 38 51 39 This is an example of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. 40 41 42 ippToPsps needs to keep pace with IPP production of new and re-processed data. Initially (2010) loading was slow. A couple of bottlenecks were: 43 - access to the DVO database is slow, especially for regions of the sky where we have high coverage (eg MD fields) 44 - ippToPsps (hopefully temporarily) performs some processing, in particular a conversion of magnitudes to fluxes and the calculation of psf, extended and cosmic ray likelihoods. Initially the likelihoods in particular were slow due to the use of an explicit database cursor, but I have since implemented a stored procedure in the MySQL database, which using an implicit cursor is much more efficient 45 52 To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log that detailed all stages of processing for a given batch. 46 53 47 54 {{{ 48 49 50 55 2011-11-29 17:20:08 | INFO | 51 56 2011-11-29 17:20:08 | INFO | New P2 batch … … 93 98 2011-11-29 17:23:46 | INFO | Attempting to publish /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz 94 99 2011-11-29 17:23:46 | INFO | Datastore publish successful 95 96 100 }}} 97 101 98 102 99 Breakdown: 103 === Breakdown === 100 104 101 reading FITS file: 0:13 102 DVO access: 2:57 103 creating Db indexes, performing all numerical manipulations (calculating fluxes, likehoods, removing duplicates, NULL fluxes etc etc): 0:13 104 creating FITS file, compressing and publishing to the datastore: 0:13 105 || '''Action''' || '''Time''' || 106 || reading the smf FITS file || 0:13 || 107 || DVO access || 2:57 || 108 || creating Db indexes, performing all numerical manipulations (calculating fluxes, likelihoods, removing duplicates, NULL fluxes etc etc) || 0:13 || 109 || creating FITS file, compressing and publishing to the datastore || 0:13 || 110 || '''Total''' || 3:38 || 105 111 106 TOTAL: 3:38 107 108 Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database. Accessing over the network slows it down substantially. Also, during this batch creation, two other clients were running on the same machine using the same MySQL database. 112 Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database (accessing over the network slows it down substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database. 109 113 110 114 … … 112 116 pre-Boston it wa taking 40 minutes oper frame. Ported all data to MySQL and access time dropped to 30 seconds. 113 117 114 implemented a multi-client version of my interface code, meaning any number can be instantiated concurrently and, by using a secure critical section, they do not interfere with one another115 118 116 119 117 Monitoring 120 == Monitoring ippToPsps == 118 121 119 Only on the IPP side of the interface do we have access to all information about all batches. From the IPP i know how many frames or stacks are available for a given survey, I know how many I have processed through my interface but I also know the progress of each item as it passed through the PSPS system:122 Only on the IPP side of the interface do we have access to all information about all batches. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system 120 123 121 124 DVO … … 128 131 there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots. 129 132 130
