Changes between Initial Version and Version 1 of addstarPV3checks

Timestamp:

Nov 24, 2015, 12:09:25 PM (11 years ago)

Author:

heather

Comment:

--

addstarPV3checks

@@ +1 @@
+This is take from Gene's email (see the bottom), and expanded.  We want to be able to verify that we have all the necessary inputs in the dvo. These are subdivided by stages:
+
+
+== cam ==
+
+I started with cam stage, and expanded upon Gene's queries, which were correct, assuming no bad quality.  
+
+=== How many do we actually expect? ===
+
+First way involves finding the cam_ids with good quality and correct label (this is how addstar does this)
+{{{
+select  count(*) , camRun.state, camProcessedExp.fault, camProcessedExp.quality from camRun join 
+camProcessedExp using (cam_id) join chipRun using (chip_id) join chipImfile using (chip_id)  where 
+camRun.label = 'LAP.PV3.20140730.final' group by state, fault, quality;
+}}}
+
+{{{
++----------+-------+-------+---------+
+| count(*) | state | fault | quality |
++----------+-------+-------+---------+
+| 22471179 | full  |     0 |       0 | 
+|    63120 | full  |     0 |    4007 | 
++----------+-------+-------+---------+
+2 rows in set (25.17 sec)
+}}}
+
+good to know. So the most number of chips expected is 22471179.  Similar query to get the number of exposures:
+
+{{{
+mysql> select  count(*) , camRun.state, camProcessedExp.fault, camProcessedExp.quality from camRun 
+join camProcessedExp using (cam_id)  where camRun.label = 'LAP.PV3.20140730.final' group by state, fault, quality; 
++----------+-------+-------+---------+
+| count(*) | state | fault | quality |
++----------+-------+-------+---------+
+|   374521 | full  |     0 |       0 | 
+|     1052 | full  |     0 |    4007 | 
++----------+-------+-------+---------+
+2 rows in set (1.74 sec)
+
+}}}
+
+second way: What happens if we add in the chip quality as well (I don't think we check this when doing addstar, I think we could addstar in chips with bad attributes, but not sure):
+{{{
+select  count(*) , camRun.state, camProcessedExp.fault, camProcessedExp.quality,chipRun.state, 
+chipProcessedImfile.fault, chipProcessedImfile.quality from camRun join camProcessedExp using 
+(cam_id) join chipRun using (chip_id) join chipImfile using (chip_id) join chipProcessedImfile using 
+(chip_id, class_id) where camRun.label = 'LAP.PV3.20140730.final' group by camRun.state, 
+camProcessedExp.fault, camProcessedExp.quality, chipRun.state, chipProcessedImfile.fault, 
+chipProcessedImfile.quality; 
+}}}
+ok, forgot that chipRun.state =!= full and there could be faults associated with cleanup.  That is less useful. I did a modified query here:
+{{{
+mysql> select  count(*) , camRun.state, camProcessedExp.fault, camProcessedExp.quality,chipRun.state, 
+chipProcessedImfile.fault, chipProcessedImfile.quality from camRun join camProcessedExp using (cam_id) 
+join chipRun using (chip_id) join chipImfile using (chip_id) join chipProcessedImfile using (chip_id, class_id) 
+where camRun.label = 'LAP.PV3.20140730.final' group by camRun.state, camProcessedExp.fault, 
+camProcessedExp.quality, chipProcessedImfile.quality;
++----------+-------+-------+---------+---------+-------+---------+
+| count(*) | state | fault | quality | state   | fault | quality |
++----------+-------+-------+---------+---------+-------+---------+
+| 22464764 | full  |     0 |       0 | cleaned |     0 |       0 | 
+|       14 | full  |     0 |       0 | cleaned |     0 |      42 | 
+|      424 | full  |     0 |       0 | cleaned |     0 |     257 | 
+|     5513 | full  |     0 |       0 | cleaned |     0 |    3012 | 
+|        6 | full  |     0 |       0 | cleaned |     0 |    3013 | 
+|    61418 | full  |     0 |    4007 | cleaned |     0 |       0 | 
+|        1 | full  |     0 |    4007 | cleaned |     0 |      42 | 
+|      813 | full  |     0 |    4007 | cleaned |     0 |     257 | 
+|      885 | full  |     0 |    4007 | cleaned |     0 |    3012 | 
++----------+-------+-------+---------+---------+-------+---------+
+9 rows in set (1 min 30.91 sec)
+}}} 
+
+-- so roughly 6000 chips (with bad chip quality but good cam quality) were probably ingested.  I don't know the right answer for this.
+
+
+=== what is currently in the dvo db ===
+
+{{{
+mysql> select camRun.label,addRun.state, count(*) from camRun join chipRun using (chip_id) join chipImfile 
+using (chip_id) join addRun on (stage_id = cam_id and stage = 'cam') where addRun.dvodb = 'LAP.PV3.20140730' 
+group by camRun.label,addRun.state;
++------------------------+-------+----------+
+| label                  | state | count(*) |
++------------------------+-------+----------+
+| LAP.PV3.20140730.final | full  | 22466679 | 
+| LAP.PV3.20140730.final | new   |     4500 | 
++------------------------+-------+----------+
+2 rows in set (29.52 sec)
+}}} 
+ok, this gets everything that is currently in stage = 'cam' with dvodb = 'LAP.PV3.20140730' - this is label independent, so if things get shifted, it's not a problem... There are 
+
+22466679 full + good, 4500 new + bad (most in a label flagged as 'investigate')
+22471179 is what we expect from the camRun+chipImfile, and it is exactly what we get if we sum up the new and the old. This is good. :)
+
+=== The tldr answer ===
+
+There are '''22471179''' chips total that we expect from the camRun, of which '''22466679''' claim to be ingested in the database.  '''4500''' failed addstar stage.
+
+I can provide a list of the chip_imfiles --- I can also provide a list of chip_imfile, minidvodb_name to help figure out what's going on ...?
+
+
+== skycal ==
+
+== fullforce ==
+
+== forcedgal ==
+
+== diff ==
+
+
+
+{{{
+I have completed the dvomerge of the slices (not yet the parallel master) for the skycal stage (stacks).  
+
+i have put together code to check the contents of the image table in a dvo db compared with a list of externID 
+(eg, stackID, warpID, camID).  I have compared cam, sky, and ff.  I find the following:
+
+1) camera stage: 
+
+I used the following sql to generate a list of chip_imfile_id which were processed by camRun under the PV3 final 
+label and then ingested via addstar into a sensible dvo:
+
+select chip_imfile_id from camRun join chipRun using (chip_id) join chipImfile using (chip_id) join addRun on 
+(stage_id = cam_id and stage = 'cam') where camRun.label = 'LAP.PV3.20140730.final' and addRun.label like 'LAP.PV3.%.cam'
+
+When I compare that to the externID values in the cam dvo database, I find 101656 chips are not in the dvo database.  
+This is probably consistent with the chips for which the astrometry was deemed to be too poor to be ingested into 
+dvo, but some spot checks would be good.  
+
+When I do the selection for all chip_imfile_ids regardless of whether addRun was applied I find 67200 more entries.  
+These are probably the one which have fault states at the camera stage, but again a double check is needed.
+
+I also find that there are still 118 duplicate chips -- there are ~2 duplicate exposures in the database which I forgot to remove.
+
+2) stack stage.  
+
+I used the following to generate a list of stackID which are in the PV3 final label and for which skycall was run:
+
+select stack_id from skycalRun join stackRun using (stack_id) where skycalRun.label = 'LAP.PV3.20140730.final'
+
+When I compare this list against the pv3.sky database, I find that all but 562 stack / skycall entries were ingested.  
+There are no duplicates
+
+3) fullforce.
+
+I used the following sql to generate a list of all warp_skyfile_ids used by fullForce runs with the PV3 final label:
+
+select warp_skyfile_id from warpRun join warpImfile using (warp_id) join fullForceResult using (warp_id) join 
+fullForceRun using (ff_id) join skycalRun using (skycal_id) join stackRun using (stack_id, skycell_id) where 
+fullForceRun.label = 'LAP.PV3.20140730.ff01'
+
+Comparing this to the image table in the ff dvo, I find 450038 warp_skyfile_id which are missing.  When heather 
+has finished the addstar runs, I can check the merge of those image tables as well.
+
+}}}