Changes between Version 1 and Version 2 of PSPSDuplicatesP2investigations

Timestamp:

Apr 15, 2014, 4:21:16 PM (12 years ago)

Author:

heather

Comment:

--

PSPSDuplicatesP2investigations

@@ -5 +5 @@
 One of the duplicates issues in PSPS occurs with duplicate objID within a P2 batch.  During ipptopsps batch creation, we remove duplicates within each OTA.  However, there are instances where there are duplicate objID that occur in 2 adjacent OTAs.
 
+== Example from a PSPS rejected batch ==
+
 One of the examples is objID 78302606220962865 in batch B01387864 - PSPS kicked this batch back to us, stating that objID 78302606220962865 was duplicated within that batch.
 
 I downloaded the batch, and investigated it. I found that the objid was in 2 adjacent OTAS (23 and 33).  If I plot ra vs objid and color the 2 OTAs differently, I get the following plots:
 
-first, a plot of ra vs dec:
+first, a plot of ra vs dec, this shows that each OTA is within certain ra dec boundaries (they do not cross)
+[[Image(radec_dupobjid.png)]]
+
+Then a plot of ra vs objID (with red on top) - the red dot in the blue section shows that one of the red objIDs crosses into blue objID territory.
+[[Image(raobjid_dupobjid_red_on_blue.png)]]
+
+A plot of ra vs objID (with blue on top) - the blue dot in the red section shows that one of the blue objIDs crosses into red objID territory.
+[[Image(raobjid_dupobjid_blue_on_red.png)]]
+
+== Tracing the duplicate ==
+
+The next step is to see if the duplicate is in the dvo, or if it is a bug in ipptopsps.  To trace it back to DVO, I did the following:
+
+(in PSPS)
+
+{{{
+select * from Object where objID = 78302606220962865
+}}}
+
+This told me which Object Batch it was in (batch_id = 239173 )
+
+Tracing it in ipptopsps
+
+{{{
+select * from batch where batch_id = 239173 
+}}}
+
+This tells me which skychunk (LAP.OB.slice09to18.OB.ipp047), and which stage_id (11847). I grepped through the log files to find out what part of the dvo catalog to look (s2230/6829.14.*), and downloaded the files and investigated them in Topcat.
+
+First, I open 6829.14.cpt - the ext_id column (the averages->extID in dvo code) is the psps objID.  I search for and find EXT_ID = 78302606220962865, and I find for that EXT_ID:
+
+||cpt table ||
+|| column name || value ||
+||EXT_ID || 78302606220962865||
+|| OBJ_ID || 165459  || 
+||CAT_ID || 118847||
+
+Then I checked the cpm file - Interestingly, it too has a EXT_ID column, but it means a different thing here (I think it is the ippObjID in PSPS, but I'm not sure).  It has CAT_ID and OBJ_ID so look for CAT_ID = 118847 (all of them, essentially, since that is specific to this dvo file), and for OBJ_ID = 165459.  This matches 14 entries in the CPM Table, which I then sorted by the TIME column. I get something like this (extraneous columns removed):
 
 
-Then a plot of ra vs objID (with red on top)
+||cpm table||
+||D_RA || D_DEC || MAG || TIME || IMAGE_ID || OBJ_ID ||CAT_ID ||EXT_ID || PHOTCODE || COMMENT || 
+||  131.64246  ||106.26292 || 17.439312 ||1282974929 ||3633028  ||165459 ||118847 ||133524686020004205 ||10302 ||  ||
+|| -1035.2087 ||-898.1283  ||17.499983 ||1282973955|| 3631787 || 165459 ||118847 ||133523559040004684 ||10304   ||||
+|| 109.75279  ||112.16554  ||18.179443 ||1282632671 ||3619266 || 165459 ||118847 ||133128554020004192 ||10402   ||||
+|| 124.10015  ||118.78707  ||18.162737 ||1282631707 ||3618073 || 165459|| 118847 ||133127438020004292 ||10402   ||||
+|| 127.0478   ||108.75634  ||17.302675|| 1277024444 ||3574104 || 165459 ||118847 ||126637550230005256 ||10223   ||||
+||  126.99159 || 108.64381 || 17.294615 ||1277023435 ||3572767 || 165459|| 118847|| 126636383230004908|| 10223   ||||
+
+||  128.88538  ||109.71859 || 17.698845|| 1276941138 ||14218296|| 165459|| 118847|| 126541131230003350|| 10123   ||  pair 1 ||
+||  0.23505946 ||-0.3748729 ||19.952366|| 1276941138 ||14218304|| 165459|| 118847|| 126541131330013760|| 10133   ||  pair 1 ||
+
+||  128.93617  ||111.362076|| 17.706528 ||1276940077 ||14219659 ||165459|| 118847 ||126539903230003240 ||10123   ||  pair 2 ||
+|| 0.25309628 ||-0.3330916 ||19.88281  ||1276940077||14219667 ||165459 ||118847 ||126539903330012937 ||10133   ||  pair 2 ||
+
+|| 126.98552  ||108.32342  ||19.573666 ||1276765061 ||14218776|| 165459 ||118847 ||126337339230002876 ||10023   ||  pair 3 ||
+||  0.3610838  ||-0.5255428|| 21.749416 ||1276765061|| 14218784 ||165459|| 118847|| 126337339330008296|| 10033   ||  pair 3 ||
+
+|| 127.135605 ||109.23181  ||18.967087 ||'''1276764112''' ||14218716|| 165459 ||118847 ||126336240230002526|| '''10023'''   ||  pair 4  || 
+|| 0.31089014 ||-0.3858239 ||21.290268 ||'''1276764112'''|| 14218724 ||165459 ||118847|| 126336240330011159 ||'''10033'''   ||  pair 4 ||
 
 
-A plot of ra vs objID (with blue on top)
-
+I highlighted the last two  (and there are 3 other pairs similar to this) . They have the same TIME, CAT_ID, and OBJ_ID, therefore I claim they are from the same exposure.  The PHOTCODE indicates that they are in XY 33 and XY 23 as well.  All of these detections (all 14) map to the averages table to get the psps objId, which causes duplicates problems because multiple detections on an exposure have the same psps objId. I am not sure the best way to remove them.