| | 22 | |
| | 23 | == list of issues with SA10 from PSPSWG with answers == |
| | 24 | |
| | 25 | * A: Object tests |
| | 26 | * A1: entries set to weird values |
| | 27 | * 1. projectionCellID: is -999 or 1226 (1226 when ndetections = -999) |
| | 28 | * a. It should be -999. When ndetections = -999 we get into funny stuff, specifically, those with ndetections = -999 are filled from skinnyObjects. Those are created from stack/detection batches. Why aren't they filled? There are a handful that are not filled, likely due to the dvo saying "oh, we don't want this as an object". The majority come from a failed batch on the psps (objid out of range) |
| | 29 | * i. Heather thinks we ought not to do skinny batches (?) |
| | 30 | * ii. if it is -999 anyways is this a column we should keep? |
| | 31 | * 2. varFlag = 0 |
| | 32 | * a. This is taken care of for future releases. It will be called qualityFlag, and is a subset of flags from infoFlag to help easily find good or bad objects |
| | 33 | * 3. surveyId = 0 |
| | 34 | * a. It should be 0 - there is a definition of surveys, and 3pi is 0. |
| | 35 | * 4. stackVer = -999 |
| | 36 | * a. This makes no sense in the current schema and Heather thinks we should remove it |
| | 37 | * 5. nw = -999 |
| | 38 | * a. We currently do not have w in the dvo or in ipptopsps |
| | 39 | * 6. consistencyFlag = -999 |
| | 40 | * a. This has been brought up and ignored before (see SA9 etc) |
| | 41 | * 7. updated = -999 |
| | 42 | * a. We don't use this flag (so then why is it there?) |
| | 43 | * A2: 12% have ra_err > 0.1 deg (same for dec_err), and 70% have ra_err = 0 (same for dec_err) |
| | 44 | * a. This needs to be investigated |
| | 45 | * A3: 0.1% have ndetections = -999 (one specific patch in ra & dec) |
| | 46 | * a. This is due to a failed batch (outside of PSPS's ra/dec range) I thought it had been fixed (conrad?) |
| | 47 | * A4: ndetections still not equal the sum of ng+nr+ni+nz+ny : far from it actually. ndetections >= 2xsum (and =2xsum for 97%) |
| | 48 | * a. This is a bug in the code, Heather knows where and how to fix |
| | 49 | * A5: sgsep: ranges from 0 to 181.28 [according to Nigel sgsep in object table is not correct, while in stackdetection it looks more normal) |
| | 50 | * a. (Gene?) |
| | 51 | * A6: ?stackdetectid: what are they? Some are negative, some have single values (equal the the filterID), and some very positive. |
| | 52 | * a. This has been fixed by Bill for future SAS |
| | 53 | * A7: General magnitudes comments: (GENE?) |
| | 54 | * 1. gMeanPSFMag - gStackPSFMag is bimodal: two peaks one at -0.03 and one at 0.12 (clearly separated) and not really the case for other bands |
| | 55 | * a. |
| | 56 | * 2. ?MeanKronMag & ?MeanKronMagerr: they span a ridiculous range (which is not the case for ?MeanPSFMag and ?MeanPSFMagerr, hence the reason to flag it) |
| | 57 | * a. |
| | 58 | * 3. ?StackPSFmagerr: covers an even more ridiculous range (- 1E59 to +1E59) |
| | 59 | * a. |
| | 60 | * B. Test on SAS10 stackdetection table with 1% of the data |
| | 61 | * B1. entries set to weird values |
| | 62 | * 1. stackTypeId is always 1 |
| | 63 | * a. suspect this is the stack type from IPP |
| | 64 | * 2. surveyID is always 0 |
| | 65 | * a. it should be because it is a 3pi like survey |
| | 66 | * 3. primaryF is always 0 |
| | 67 | * a. we don't set it (should we? ) |
| | 68 | * 4. x/yposerr: -999 to 17k while x/ypos 10 to 6.3k |
| | 69 | * a. |
| | 70 | * 5. raerr & decerr: quantized in 0.25 steps + in silly units (not the same as xposerr)+ negative from time to time (not just -999) + range is huge (-8000 to 8000) |
| | 71 | * a. |
| | 72 | * 6. apFluxerr: -999 (i.e. never set) |
| | 73 | * a. |
| | 74 | * 7. kronrad: min value ~5 : is this expected? |
| | 75 | * a. |
| | 76 | * 8. psfchisq: 1k to 5k typically.... peak at 2.8k: what's the meaning of that number |
| | 77 | * a. |
| | 78 | * 9. activeFlag: always 0 |
| | 79 | * a. we already know about it, if we don't use it can we drop it |
| | 80 | * 10. assocDate: what does it correspond to |
| | 81 | * a. |
| | 82 | * 11. historyModNum: always 0 |
| | 83 | * a. not used |
| | 84 | * 12. datarelease: 10 (ok I presume, as it corresponds to SAS10) |
| | 85 | * a. yes |
| | 86 | * B2. ProjectionCellID do not match between stackdetection and object: not a single one match! Which one is the correct one? |
| | 87 | * a. stackdetection is the correct one - we have the column in Object but we do not use it |
| | 88 | * B3. sgsep do not match between stackdetection and object [see above]; Nigel has the feeling sgsep in stackdetection behaves more as expected. |
| | 89 | * a. The object one comes from the dvo averages table. May need Gene to explain |
| | 90 | * B4. projectionCellID == skycellID: is this normal or is this an artefact of SAS10 being small? |
| | 91 | * a. This is a known issue and not to be resolved yet (why?) |
| | 92 | * B5. psflikelihood: why <0 for 10% of the stackdetections and -999 for ~25% of the stackdetections |
| | 93 | * a. ? |
| | 94 | * C. PSPS flags: |
| | 95 | * C1. objectxflags: what does ID_SECF_HAS_STACK correspond to: forced photometry on the stack if set? Not clear why it should indicate whether there is a stack photometry as if there isn't any, then the corresponding photometry column should be set to -999 |
| | 96 | * a. ? |
| | 97 | * D. PSI/PSPS suggestions: |
| | 98 | * D1. It would be good if a small command could be defined for this expression int_ran = (ippobjid % 10) *1000 + ((objid / 10000) % 10)*100 + ((ippobjid / 10) % 10)*10 +(objid % 10) . The neat part of int_ran is that it runs from 0 to 9999 and is randomly distributed. It can be selected from nearly all tables directly in a repeatable fashion. int_ran is very quick to compute (barely increases the runtime of the query). Equally important (if not key), one can match tables which were selected on the same int_ran condition, making it far superior than any true random number associated with any table. The only flaw is that one cannot select any fraction smaller than 1/10000, i.e. 0.01%, but I believe this will rarely be of interest. |
| | 99 | * a. Conrad? |
| | 100 | * D2. Using the query builder on IPP to select all entries in the object table result in a crazy column layout: the first few columns end up being o_objID, o_b, o_ni, o_yMeanPSFMagErr, o_gFlags,.... which is clearly unpractical. This doesn't happen with PSVO, which lists all columns according to the schema browser layout (i.e. standard one). |
| | 101 | * a. Conrad? |
| | 102 | * More Issues: |
| | 103 | * 1. Matching stackdetection and object: for a few objects matched on objid, they don't have the same ippobjid (<0.1%) [found this even on a 10% random sampling, because the random number uses ippobjid and objid] |
| | 104 | * a. |
| | 105 | * 2. select * from Detectionflags order by value: query does not return the same flags as listed on PSPSflags_sas10 |
| | 106 | * a. this is on heather's list to be fixed |
| | 107 | * 3. objectflags and stackdetectionflags tables are inexistent, i.e. one can not retrieve from the database the meaning of the flags. This should be contrasted with: select * from detectionflags for the detection table |
| | 108 | * a. this is on heather's list to be fixed |
| | 109 | * 4. Object table flags: |
| | 110 | {{{ |
| | 111 | -1. If stackphotometry exist (xFlags & 0x20 !=0), why does xStackKronMag or xStackPSFMag == -999? |
| | 112 | We have for g r & i: |
| | 113 | 0.5%, 8.5% in g (PSF, Kron) |
| | 114 | 0.7%, 5.5% in r (PSF, Kron) |
| | 115 | 0.7%, 4.5% in i (PSF, Kron) |
| | 116 | - 2. What is the difference between: |
| | 117 | - ID_OBJ_EXT (in object.infoflag) and ID_PSPS_OBJ_EXT (in object.xflags) ? |
| | 118 | - ID_SECF_USE_SYNTH and ID_SECF_USE_UBERCAL in object.xflags? |
| | 119 | - 3. Object table flags, either in object.objinfoflags or in object.xflags, do not list flags for the following information: |
| | 120 | - whether the object is detected on the stack or on individual frame. |
| | 121 | The flag xflags.ID_SECF_HAS_STACK only tells whether the object has a stack photometry or not |
| | 122 | and as mentionned above a fair fraction of objects with the flag set do not have proper stack photometry (-999). |
| | 123 | - whether the stack photometry for the object is forced or not per band |
| | 124 | - whether quantites come from stack or individual frame (e.g. ra & dec) |
| | 125 | - wheteher photometry was attempted or not (assuming Kron and PSF are automatically done at the same time; |
| | 126 | otherwise one need to have a flag telling whether psfphotometry / kronphotometry was attempted or not) |
| | 127 | - 4. Object table need to list the number of detections vs the number of times a position was looked at (for a given survey). This is an issue for non-detection / false positives. Additionally the number of detections should not just be the sum of all detections scrambled together, but either the number of detection or the number of stack detections. In an ideal world, different columns for ndetections and nstackdetections should be listed. |
| | 128 | - 5. The provenance of many flags in the object table is not clear: |
| | 129 | - where does xflags.ID_PSPS_OBJ_EXT come from? |
| | 130 | - where does xflags.ID_OBJ_GOOD come from? |
| | 131 | - 6. Unclear whether the following flags are of much use in object.xflags (poor documentation might be a reason to fall in this category): |
| | 132 | ID_SECF_STAR_FEW = 0x00000001, // used within relphot: skip star |
| | 133 | ID_SECF_STAR_POOR = 0x00000002, // used within relphot: skip star |
| | 134 | ID_SECF_USE_SYNTH = 0x00000004, // synthetic photometry used in average measurement |
| | 135 | ID_SECF_USE_UBERCAL = 0x00000008, // synthetic photometry used in average measurement |
| | 136 | ID_SECF_HAS_PS1 = 0x00000010, // PS1 photometry used in average measurement |
| | 137 | ID_PHOTOM_PASS_0 = 0x00000100, // average magnitude calculated in 0th pass |
| | 138 | ID_PHOTOM_PASS_1 = 0x00000200, // average magnitude calculated in 1th pass |
| | 139 | ID_PHOTOM_PASS_2 = 0x00000400, // average magnitude calculated in 2th pass |
| | 140 | ID_PHOTOM_PASS_3 = 0x00000800, // average magnitude calculated in 3th pass |
| | 141 | ID_PHOTOM_PASS_4 = 0x00001000, // average magnitude calculated in 4th pass |
| | 142 | ID_SECF_OBJ_EXT = 0x01000000, // DVO location of "extended in this band" For PSPS this bit is moved |
| | 143 | - 7. Unclear whether the following flags are of much use in object.objinfoflags |
| | 144 | (poor documentation might be a reason to fall in this category; note that I have remove all those not any longer set): |
| | 145 | ID_STAR_FEW = 0x00000001, // used within relphot: skip star |
| | 146 | ID_STAR_POOR = 0x00000002, // used within relphot: skip star |
| | 147 | ID_OBJ_EXT_ALT = 0x02000000, // extended in external data (eg, 2MASS) |
| | 148 | ID_OBJ_GOOD_ALT = 0x08000000, // good-quality measurement in external data (eg, 2MASS) |
| | 149 | }}} |
| | 150 | |
