ReadStarsSDSS does not associate measurements into objects

[Sebastian Jester]

This ticket is the duplicate of a ps-ipp-users email for easier reference.

Executive summary: ReadStarsSDSS does not produce dvo catalogs where every object has a measurement in all of ugriz, but 5 times as many objects, each with only one out of ugriz set. Moreover, I haven't been able to pull out the original magnitudes nor errors from the dvo catalog. So ReadStarSDSS currently does not produce useful dvo catalogs (well, not being able to pull out errors is perhaps due to my lack of knowledge about dvo)

Instead, I think SDSS (and UKIDSS, too) need functions analogous how 2MASS is read, i.e. like get2mass_3star_full() from Ohana/src/addstar/src/get2mass_full.c which gets called by load2mass_as_rawdata() in the eponymous .c file which gets called from main() in load2mass.c

The other conclusion I have is that to load SDSS objects properly without differences between the bands, we'd need a 'sec' photcode for u.

I have a dvo.photocode file on a branch that does that, as well as a hacked version of ReadStarsSDSS, on a branch:
​https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/branches/sj_branches/sj_SDSSaddstar_branch_r23928_20090420/ (or see browser:/branches/sj_branches/sj_SDSSaddstar_branch_r23928_20090420/​ )
(just copied Ohana in there on the date and rooted at the revision given, and ippconfig just now, i.e. the branch has what you need for using dvo standalone.)

Here is the long version:

After addstar on a tsObj file via ReadStarsSDSS, the measurements from the different filters do not seem to get associated into a single object. Both in mextract and avextract, the dvo database contains 5 objects for each SDSS object, with nans everywhere except in one filter per object.

I used av/mextract -region RA DEC U_SDSS:inst U_SDSS:err G_SDSS:inst G_SDSS:err R_SDSS:inst R_SDSS:err I_SDSS:inst I_SDSS:err Z_SDSS:inst Z_SDSS:err

avextract produces:
...
352.883087 -1.276238 22.048 nan nan nan nan nan nan nan nan nan
352.883087 -1.276235 nan nan nan nan nan nan nan nan 22.469 nan
352.883118 -1.276250 nan nan nan nan 20.902 nan nan nan nan nan
352.883240 -1.276169 nan nan nan nan nan nan 20.740 nan nan nan
352.883270 -1.276260 nan nan 21.338 nan nan nan nan nan nan nan
...

mextract produces weird numbers:
...
352.883087 -1.276238 -56.859 nan nan nan nan nan nan nan nan nan
352.883087 -1.276235 nan nan nan nan nan nan nan nan -56.439 nan
352.883118 -1.276250 nan nan nan nan -58.006 nan nan nan nan nan
352.883240 -1.276169 nan nan nan nan nan nan -58.167 nan nan nan
352.883270 -1.276260 nan nan -57.570 nan nan nan nan nan nan nan
352.883514 -1.206632 nan nan -56.872 nan nan nan nan nan nan nan
...

Both produce identical results before and after relphot -averages -update, i.e., relphot has no effect on the catdir produced by ReadStarsSDSS.

Also note that all the errors are nan! I.e. I don't seem to be able to pull out the original errors anywhere (this goes back to another discussion on the list where Michael last responded).

The tables were generated by the attached dvo script. To reproduce, you need
​http://das.sdss.org/imaging/1056/40/calibChunks/1/tsObj-001056-1-40-0192.fit

ADDSTAR_RADIUS in sdssmosaic/dvo.config is 1.

The examples above are with my branched version of Ohana
​https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/branches/sj_branches/sj_SDSSaddstar_branch_r23928_20090420/Ohana
and the ippconfig from svn head, except that I used my own dvo.photcodes file (also attached) which declares 'u' as a 'sec' photcode and links U_SDSS to it. The only real change in ReadStarsSDSS I made is that I skip over objects whose photons are also contained in other objects (i.e. I include only 'primary' and 'secondary' objects - see extensive notes in ReadStarsSDSS.c on that branch).

However, I also tried the Ohana tarball Gene had made for distribution, Ohana.20081024; there have been extensive changes to find_matches_closest.c between that tarball and svn head, but I still get only one non-nan number per row.

Come to think of it, it must be that ReadStarsSDSS() adds the measurements in the 5 SDSS bands as 5 separate stars, and then find_matches_closest() doesn't match the list to itself, does it? So dvo never finds out that there are bunches of 5 objects that are very close to each other, and considers them as separate objects, all with measurements only in 1 out of the 5 bands.

Associations *would* happen if I loaded another epoch of SDSS data, but you won't be able to predict which of the 5 new bands gets associated with which of the old bands, so you'd have some objects with gg some with gr some with iz etc. That's clearly not what's desired.

The same thing will happen with UKIDSS data or any other multi-band survey. Like 2MASS, in fact, which dvo does read and associate properly so colours can be computed, doesn't it? It seems that the 2MASS loader goes a different route from the normal addstar, in that it does the matches in find_matches_refstars() while ReadStarsSDSS calls find_matches_closes().

To me, it looks like the fix (and the way to read UKIDSS data) is to write something analogous to the 2MASS loaders. What is the deeper reason for adding the SDSS data the way they are done?

[Sebastian Jester]

Since I can't create attachments (see #1226) see ​http://pan-starrs.ifa.hawaii.edu/mailman/private/ps-ipp-dev/msg02720.html for the attachments

[Sebastian Jester]

See changesets [24982] and [24981] which add copious comments in ReadStarsSDSS.c on the browser:/branches/sj_branches/sj_SDSSaddstar_branch_r23928_20090420/​ branch.

[eugene]

This was an error in the order of events. This was fixed by defining a grouping of 5 when loading the SDSS data (so find_matches would know to apply all 5 detections to the same object).

fixed in r27392