Investigation of P2 duplicates issues

Introduction

One of the duplicates issues in PSPS occurs with duplicate objID within a P2 batch. During ipptopsps batch creation, we remove duplicates within each OTA. However, there are instances where there are duplicate objID that occur in 2 adjacent OTAs.

Example from a PSPS rejected batch

One of the examples is objID 78302606220962865 in batch B01387864 - PSPS kicked this batch back to us, stating that objID 78302606220962865 was duplicated within that batch.

I downloaded the batch, and investigated it. I found that the objid was in 2 adjacent OTAS (23 and 33). If I plot ra vs objid and color the 2 OTAs differently, I get the following plots:

first, a plot of ra vs dec, this shows that each OTA is within certain ra dec boundaries (they do not cross)

Then a plot of ra vs objID (with red on top) - the red dot in the blue section shows that one of the red objIDs crosses into blue objID territory.

A plot of ra vs objID (with blue on top) - the blue dot in the red section shows that one of the blue objIDs crosses into red objID territory.

Tracing the duplicate

The next step is to see if the duplicate is in the dvo, or if it is a bug in ipptopsps. To trace it back to DVO, I did the following:

(in PSPS)

select * from Object where objID = 78302606220962865

This told me which Object Batch it was in (batch_id = 239173 )

Tracing it in ipptopsps

select * from batch where batch_id = 239173 

This tells me which skychunk (LAP.OB.slice09to18.OB.ipp047), and which stage_id (11847). I grepped through the log files to find out what part of the dvo catalog to look (s2230/6829.14.*), and downloaded the files and investigated them in Topcat.

First, I open 6829.14.cpt - the ext_id column (the averages->extID in dvo code) is the psps objID. I search for and find EXT_ID = 78302606220962865, and I find for that EXT_ID:

cpt table
column name	value
EXT_ID	78302606220962865
OBJ_ID	165459
CAT_ID	118847

Then I checked the cpm file - Interestingly, it too has a EXT_ID column, but it means a different thing here (I think it is the ippObjID in PSPS, but I'm not sure). It has CAT_ID and OBJ_ID so look for CAT_ID = 118847 (all of them, essentially, since that is specific to this dvo file), and for OBJ_ID = 165459. This matches 14 entries in the CPM Table, which I then sorted by the TIME column. I get something like this (extraneous columns removed):

cpm table
D_RA	D_DEC	MAG	TIME	IMAGE_ID	OBJ_ID	CAT_ID	EXT_ID	PHOTCODE	COMMENT
131.64246	106.26292	17.439312	1282974929	3633028	165459	118847	133524686020004205	10302
-1035.2087	-898.1283	17.499983	1282973955	3631787	165459	118847	133523559040004684	10304
109.75279	112.16554	18.179443	1282632671	3619266	165459	118847	133128554020004192	10402
124.10015	118.78707	18.162737	1282631707	3618073	165459	118847	133127438020004292	10402
127.0478	108.75634	17.302675	1277024444	3574104	165459	118847	126637550230005256	10223
126.99159	108.64381	17.294615	1277023435	3572767	165459	118847	126636383230004908	10223

128.88538	109.71859	17.698845	1276941138	14218296	165459	118847	126541131230003350	10123	pair 1
0.23505946	-0.3748729	19.952366	1276941138	14218304	165459	118847	126541131330013760	10133	pair 1

128.93617	111.362076	17.706528	1276940077	14219659	165459	118847	126539903230003240	10123	pair 2
0.25309628	-0.3330916	19.88281	1276940077	14219667	165459	118847	126539903330012937	10133	pair 2

126.98552	108.32342	19.573666	1276765061	14218776	165459	118847	126337339230002876	10023	pair 3
0.3610838	-0.5255428	21.749416	1276765061	14218784	165459	118847	126337339330008296	10033	pair 3

127.135605	109.23181	18.967087	1276764112	14218716	165459	118847	126336240230002526	10023	pair 4
0.31089014	-0.3858239	21.290268	1276764112	14218724	165459	118847	126336240330011159	10033	pair 4

I highlighted the last two (and there are 3 other pairs similar to this) . They have the same TIME, CAT_ID, and OBJ_ID, therefore I claim they are from the same exposure. The PHOTCODE indicates that they are in XY 33 and XY 23 as well. All of these detections (all 14) map to the averages table to get the psps objId, which causes duplicates problems because multiple detections on an exposure have the same psps objId. I am not sure the best way to remove them.