IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 625 and Version 626 of MD.GR0


Ignore:
Timestamp:
Feb 6, 2013, 3:03:57 PM (13 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • MD.GR0

    v625 v626  
    1212
    1313Where to find the data:
    14  * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR/  (full: pixel images of reprocessed nightly data products like nightly stacks)
     14 * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR/  (full: pixel images of reprocessed nightly data products like night stacks)
    1515 * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR-cat/  (cleaned: catalogs of reprocessed nightly data products like camera SMFs)
    1616 * http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/  (full: pixel images for updated refstacks and reftest stacks)
    … …  
    3232 * ''12/31/11'' MD06 finally finished, distributed.
    3333 
    34  * ''1/09/12'' MD08, 09 on hold while finishing up full reprocessing and release of V3 products now (i.e., nightly stacks, SMFs, SSdiffs).
     34 * ''1/09/12'' MD08, 09 on hold while finishing up full reprocessing and release of V3 products now (i.e., night stacks, SMFs, SSdiffs).
    3535 * ''1/11/12'' MD06,04 full camera SMFs on datastore, MD07 later today. MD04 set has some duplicates with both unmagicked SMFs (data_group=MD04.GR0, sub-sample of ~150/filter best seeing) and magicked SMFs (data_group=condor_MD04.V3_01_dg, full sample of ~400/filter loaded into PSPS Nov. 2011)
    36  * ''1/11'' full V3 nightly stack reprocessing with MD06 started. -- important note: these are now being run allowing 2 or more inputs to make a stack (down from 4 during the magic era) so outer skycells may have strange behavior that we need feedback on.
    37  * ''1/17'' majority of MD06 full reprocessing of nightly stacks on datastore. MD04 full V3 nightly stack reprocessing started.
    38  * ''1/23'' MD04.V3 full reprocessing of nightly stacks finished and pushed out to datastore.
     36 * ''1/11'' full V3 night stack reprocessing with MD06 started. -- important note: these are now being run allowing 2 or more inputs to make a stack (down from 4 during the magic era) so outer skycells may have strange behavior that we need feedback on.
     37 * ''1/17'' majority of MD06 full reprocessing of night stacks on datastore. MD04 full V3 night stack reprocessing started.
     38 * ''1/23'' MD04.V3 full reprocessing of night stacks finished and pushed out to datastore.
    3939 * ''4/17'' MD08.V3 restarted from scratch after raid crash in November/December with the new ipp-20120404 tag that will include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map, but not boost factors).
    4040 * ''4/25'' MD08.GR0 reprocessed and distributed, refstacks made and distributed, staticsky run and distributed.
    4141 * ''6/01'' MD09.GR0 chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71.
    42  * ''6/03'' MD09 g,y refstack started, nightly stack reprocessing running.
    43  * ''6/04'' MD09 z refstack started. g,y refstack and reftests and nightly stack pushed to distribution.
    44  * ''6/06'' MD09 r,i refstacks started. z refstack, 90% nightly stacks out to datastore.
    45  * ''6/07'' MD09 refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and nightly stacks that are possible distributed.
     42 * ''6/03'' MD09 g,y refstack started, night stack reprocessing running.
     43 * ''6/04'' MD09 z refstack started. g,y refstack and reftests and night stack pushed to distribution.
     44 * ''6/06'' MD09 r,i refstacks started. z refstack, 90% night stacks out to datastore.
     45 * ''6/07'' MD09 refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and night stacks that are possible distributed.
    4646 * ''6/18''  MD07 (pre-ubercal) staticsky run attempted, see notes below
    4747 * "7/05" new MD10 refstack started, see [wiki:MD10.refstack.20120705]
    48  * "7/29" MD09.GR0 XY67,71 chip problem (~48 exposures) not fixable so set to poor quality, remaining camera SMF and nightly stacks processed and distributed.
     48 * "7/29" MD09.GR0 XY67,71 chip problem (~48 exposures) not fixable so set to poor quality, remaining camera SMF and night stacks processed and distributed.
    4949 * "8/03" new MD01 refstack started, like new MD10 more of continuation of GR0 (separate wiki pages) -- [wiki:MD01.refstack.20120803]
    5050 * 08/31 MD09.refstack.20120603 replaced with MD09.refstack.20120831 (run with ipp-20120802 tag) to fix the weight pixel arc region of lower sensitivity
    … …  
    7474   * MD10.refstack.20110722 y-band skycell.048 had rejection problem, fixed with MD10.refstack.20110821
    7575 * MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims
    76  * with magic turned off now (01/2012), MD04+MD06 have 2+ warp inputs for nightly stack creation to test how well boundaries come out and has fault/quality issue with 2-3 input cases showing some inputs have incorrect PSF size values measured/passed. rare (~<1%) but should be looked into (ie stack_id=609561)
     76 * with magic turned off now (01/2012), MD04+MD06 have 2+ warp inputs for night stack creation to test how well boundaries come out and has fault/quality issue with 2-3 input cases showing some inputs have incorrect PSF size values measured/passed. rare (~<1%) but should be looked into (ie stack_id=609561)
    7777 * adding a quality column to ippMonitor for evaluating missing stack skycells would be useful to have on ''stack processed skyfile''
    7878 
    … …  
    102102   * [wiki:MD.GR0#MD09.GR0 MD09.GR0]: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible.
    103103 * All fields should be looked at to see if there are any problems with the new tessellation and dither pattern.
    104  *  [wiki:MD.GR0#MD07.GR0 MD07], [wiki:MD.GR0#MD09.GR0 MD09] will have a preliminary nightly stack made for testing the revised center as well as [wiki:MD.GR0#MD06.GR0 MD06] for NGC4258.
     104 *  [wiki:MD.GR0#MD07.GR0 MD07], [wiki:MD.GR0#MD09.GR0 MD09] will have a preliminary night stack made for testing the revised center as well as [wiki:MD.GR0#MD06.GR0 MD06] for NGC4258.
    105105
    106106
    … …  
    110110Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking.
    111111 * Magic destreaking: a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.
    112  * Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks) and also a fairly balanced dither set. better with more recent data to limit AGN/variable star variations.
     112 * Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over night stacks) and also a fairly balanced dither set. better with more recent data to limit AGN/variable star variations.
    113113 
    114114Deep stacks -- for deep static sky science.
    … …  
    147147
    148148== Stacks ==
    149 __Night Stacks__: All nights reprocessed and stacked into a nightly stack using the current nightly stack configuration used for nightly science nightly stacks when processed, including reference catalog
     149__Night Stacks__: All nights reprocessed and stacked into a night stack using the current night stack configuration used for nightly science night stacks when processed, including reference catalog
    150150 * should attempt to make stack with 2 or more input warps now
    151  * some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the nightly stack.
     151 * some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the night stack.
    152152 * the sample of stacks can be a mix of synthetic/ubercal catalog calibrations but NOT for the warps going into the stack itself.
    153153
    … …  
    168168
    169169== Stack Photometry (Static Sky + Skycal) ==
    170 Static sky (stack) photometry and recalibration (skycal) is run on refstacks, deepstacks and nightly stacks. In trying to run as a uniform group, will keep a summary of info here, but details like specific faults, poor quality etc will be in the respective MD section. For example some skycells can have a clear offset fix to the zeropoint, not actually set to 25.0 in stack pixels, sometimes the majority of the field is off from zeropoint of 25.  There appears to be upwards of 3-5% error likely in the photometry across the FPA (see [wiki:MD.GR0#MD09.GR0]) even in the recalibration of the stack photometry (skycal stage).
     170Static sky (stack) photometry and recalibration (skycal) is run on refstacks, deepstacks and night stacks. In trying to run as a uniform group, will keep a summary of info here, but details like specific faults, poor quality etc will be in the respective MD section. For example some skycells can have a clear offset fix to the zeropoint, not actually set to 25.0 in stack pixels, sometimes the majority of the field is off from zeropoint of 25.  There appears to be upwards of 3-5% error likely in the photometry across the FPA (see [wiki:MD.GR0#MD09.GR0]) even in the recalibration of the stack photometry (skycal stage).  A much worse example is seen in [wiki:MD01.refstack.20120803].
    171171
    172172 * '''recalibration of the stack photometry catalogs has no method for distribution''' -- email for access
    … …  
    303303 * (I) incomplete -- for MD03, 05 stack photometry stalled while data disks being shuffled..
    304304 * (mixed)
    305   * finish reprocessing all exposures to V3 for warp/nightly stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing) -- finished 12/30/13
     305  * finish reprocessing all exposures to V3 for warp/night stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing) -- finished 12/30/13
    306306  * (mixed) -- both camera/SMF and night stacks can have calibrations based on the synthetic catalogs (initial sample run) and ubercal catalogs (finish V3 run).  See specific section for detail.
    307307  * (mixed+full) -- MD05 -- some older and newer processing, so mix of magic and photometry catalog used and then a re-reprocessing of MD05 was done on the exposures originally processed with synthetic catalog. See [wiki:MD.GR0#MD05.GR0extrareprocessedsetwithallubercalcatalog]
    … …  
    310310  * iq/psf stack on hold until MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time.
    311311  * deep stack for MD04,09 for MD.GR0 only, all others on hold until MD.GR1 (many many updates would be necessary..).
    312  * reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete nightly stacks)
     312 * reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete night stacks)
    313313 
    314314
    … …  
    338338
    339339
    340 MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent nightly stack set of exposures. This may need to be shifted in the next iteration.
     340MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent night stack set of exposures. This may need to be shifted in the next iteration.
    341341
    342342|| [[Image(MD10_g_smf.png,150px)]] || [[Image(MD10_r_smf.png,150px)]] || [[Image(MD10_i_smf.png,150px)]] ||
    … …  
    449449
    450450 
    451 || Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical nightly stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version. || [[Image(MD10.20110814_i_testdeep.png,300px)]] ||
    452 
    453 
    454 || A representative skycell mapping of the new tessellation and good_fraction coverage. Nightly stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087. || [[Image(MD10.refstack.20110821.NULL.g.00000.stksummap.png,300px)]] ||
     451|| Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical night stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version. || [[Image(MD10.20110814_i_testdeep.png,300px)]] ||
     452
     453
     454|| A representative skycell mapping of the new tessellation and good_fraction coverage. Night stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087. || [[Image(MD10.refstack.20110821.NULL.g.00000.stksummap.png,300px)]] ||
    455455
    456456Other plots that may be good to see
    … …  
    484484
    485485Planned distribution of data products:
    486  * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, nightly stack, warp-stack and stack-stack diffims
     486 * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, night stack, warp-stack and stack-stack diffims
    487487 * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR-cat (cleaned): camera, warp-stack and stack-stack diffims
    488488
    … …  
    491491Available data products on the datastore:
    492492 * warp (full): done
    493  * nightly stack (full): done
     493 * night stack (full): done
    494494 * warp-stack diffim (full, cleaned): done, done
    495495 * camera/SMF: done
    … …  
    508508Summary of any failed data products during processing.
    509509
    510  * two skycell.021(and?) failed to produce a nightly stack (date?)
     510 * two skycell.021(and?) failed to produce a night stack (date?)
    511511
    512512
    … …  
    542542
    543543''Status:''
    544  * Refstack and sample nightly stacks finished and distributed.
     544 * Refstack and sample night stacks finished and distributed.
    545545 * 08/2012: to be replaced with [wiki:MD01.refstack.20120803]
    546546
    547 Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.
     547Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as night stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.
    548548
    549549|| MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot. || [[Image(MD01_g_smf.png,200px)]] ||
    … …  
    615615 * dist_group: MD01.GR
    616616
    617 Distributed under ''ps1-md-GR'' -- currently only a small sub-sample of nightly stack images only. SSdiff sample to be processed still.
     617Distributed under ''ps1-md-GR'' -- currently only a small sub-sample of night stack images only. SSdiff sample to be processed still.
    618618
    619619Data volume on disk:
    … …  
    628628 * data_group: MD01.GR0.20121122
    629629
    630 Nightly stack for all filters grizy::
     630Night stack for all filters grizy::
    631631 * label: MD01.GR0
    632632 * data_group: MD01.GR0.20121122.nightstack.<obsdate>
    … …  
    638638
    639639''Status:''
    640  * Refstack and sample nightly stacks finished and distributed.
     640 * Refstack and sample night stacks finished and distributed.
    641641 * 10/5/2012: to be replaced with [wiki:MD02.refstack.20120927] -- diffims 20121001 onwards will use.
    642642
    643 Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures to release nightly stacks as well. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.
    644 
    645 || MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell. While warps will extend to some of the outer skycells, the nightly stack distribution will define what is likely for the skycells released due to magic. 012-017, 21-28, 31-38, 41-48, 51-58, 61-68, 71-78, 82-87  ||  [[Image(MD02_g_smf.png,200px)]] ||
     643Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures to release night stacks as well. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.
     644
     645|| MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell. While warps will extend to some of the outer skycells, the night stack distribution will define what is likely for the skycells released due to magic. 012-017, 21-28, 31-38, 41-48, 51-58, 61-68, 71-78, 82-87  ||  [[Image(MD02_g_smf.png,200px)]] ||
    646646
    647647Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:
    … …  
    709709 * dist_group: MD02.GR
    710710
    711 Distributed under ''ps1-md-GR'' -- sample on nightly stack images only. SSdiff sample still to be run.
     711Distributed under ''ps1-md-GR'' -- sample on night stack images only. SSdiff sample still to be run.
    712712
    713713A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data
    … …  
    730730 * data_group: MD02.GR0.20121210
    731731
    732 Nightly stack for all filters grizy::
     732Night stack for all filters grizy::
    733733 * label: MD02.GR0
    734734 * data_group: MD02.GR0.20121210.nightstack.<obsdate>
    … …  
    741741----
    742742= MD03.GR0 =
    743 ''Status:'' refstacks and sample nightly stacks finished and distributed
    744  * 10/3: Sub-sample of y and z-band exposures through warp, sample nightly stacks made. y-band refstack in progress.
     743''Status:'' refstacks and sample night stacks finished and distributed
     744 * 10/3: Sub-sample of y and z-band exposures through warp, sample night stacks made. y-band refstack in progress.
    745745 * 10/4: r,i through warp. z-band refstack in progress
    746746 * 10/6: g through warp, ri refstack in progress
    747  * 10/8: all refstacks queued for distribution, nightly stack sample as well.
     747 * 10/8: all refstacks queued for distribution, night stack sample as well.
    748748 * 11/07/2012: to be replaced with [wiki:MD03.refstack.20121101] -- diffims 20121026 onwards will use this
    749749
    … …  
    823823 * dist_group: MD03.GR
    824824
    825 Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.
     825Distributed under ps1-md-GR -- sample of night stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.
    826826
    827827Data volume on disk:
    … …  
    837837 * data_group: MD03.GR0.20121212
    838838
    839 Nightly stack for all filters grizy::
     839Night stack for all filters grizy::
    840840 * label: MD03.GR0
    841841 * data_group: MD03.GR0.20121212.nightstack.<obsdate>
    … …  
    851851= MD04.GR0 =
    852852
    853 ''Status:'' Refstacks and sample nightly stacks finished and distributed. Very deep stack tests in progress. Full reprocessing of all possible nightly stacks finished and to datastore.
     853''Status:'' Refstacks and sample night stacks finished and distributed. Very deep stack tests in progress. Full reprocessing of all possible night stacks finished and to datastore.
    854854 * 10/2 Starting with i-band rather than y for the sub-sample reprocessing exposures through warp.
    855855 * 10/4 z-band through warp done. testing very deep >150 stack on i-band.  r-band running through warp
    … …  
    859859 * 10/9: r-band running, g-band running
    860860 * 10/10: remaining y refstack finishing up
    861  * 10/16: refstack and sample nightly stack set finished and distributed
    862  * 01/17/12: full reprocessing of all possible nightly stacks in V3 started (finished 1/23 and to datastore)
     861 * 10/16: refstack and sample night stack set finished and distributed
     862 * 01/17/12: full reprocessing of all possible night stacks in V3 started (finished 1/23 and to datastore)
    863863 * 11/25/2012: refstack to be replaced with [wiki:MD04.refstack.20121125] -- diffims onwards will use this
    864864
    … …  
    955955== MD04.GR0 reprocessed sample ==
    956956
    957 Initial set of chip->warp (including camera) and nightly stack sample:
     957Initial set of chip->warp (including camera) and night stack sample:
    958958 * label: MD04.GR0
    959959 * dist_group: MD04.GR
    … …  
    965965
    966966Distributed on datastore under
    967  * ps1-md-GR: sample of nightly stack images only. SSdiff sample still to be run.
     967 * ps1-md-GR: sample of night stack images only. SSdiff sample still to be run.
    968968 * ps1-md-GR-cat: unmagicked camera SMFs with data_group MD04.GR0 released (dupe to the full magicked SMF release from condor processing that was magicked for PSPS loading Nov. 2011)
    969969
    … …  
    990990
    991991= MD05.GR0 =
    992 ''Status:'' Refstacks and sample nightly stacks finished and distributed.
     992''Status:'' Refstacks and sample night stacks finished and distributed.
    993993 * 10/7: y-band chip-warp setup.
    994994 * 10/8: yzi chip-warp run
    995995 * 10/9: rg chip-warp run, yzi sample night stacks setup
    996  * 10/10: sample nightly stacks yzi finished, gr started, y refstack started
    997  * 10/16: refstack and sample nightly stack set finished and distributed
     996 * 10/10: sample night stacks yzi finished, gr started, y refstack started
     997 * 10/16: refstack and sample night stack set finished and distributed
    998998 * 10/05/2012: restarted processing remaining exposures into V3 -- will be somewhat a mess
    999  * 10/13: nightly stacks for remaining exposures finished (important to see details below).
     999 * 10/13: night stacks for remaining exposures finished (important to see details below).
    10001000 * 12/06/12: new refstack [wiki:MD05.refstack.20121202] will replace this one for diffims in the 2012/2013 season
    10011001 
    … …  
    10621062
    10631063== MD05.GR0 reprocessed set ==
    1064 Initial sample set of chip-warp and sub-sample nightly stack reprocessing:
     1064Initial sample set of chip-warp and sub-sample night stack reprocessing:
    10651065 * label: MD05.GR0
    10661066 * data_group: MD05.GR0.20111007 = y
    … …  
    10711071 * dist_group: MD05.GR
    10721072
    1073 Finished nightly stacks with past chip->warp reprocessing from 20111007,08,09,10,11 (pixels calibrated with '''synth catalog'''):
     1073Finished night stacks with past chip->warp reprocessing from 20111007,08,09,10,11 (pixels calibrated with '''synth catalog'''):
    10741074 * label: MD05.GR0
    10751075 * data_group: MD05.GR0.201210xx.nightstack.<obsdate> = grizy
    … …  
    10791079 * data_group: MD05.GR0.20121005 = grizy
    10801080
    1081 Nightly stacks with new chip->warp reprocessing (IPP tag ipp-20120802 and pixels calibrated with ubercal catalog)
     1081Night stacks with new chip->warp reprocessing (IPP tag ipp-20120802 and pixels calibrated with ubercal catalog)
    10821082 * label: MD05.GR0
    10831083 * data_group: MD05.GR0.20121005.nightstack.<obsdate> = grizy
    10841084
    1085 Nightly stack faults:
     1085Night stack faults:
    10861086{{{
    10871087select stack_id,label,data_group,fault,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and fault>0;
    … …  
    11061106}}}
    11071107
    1108 Nightly stack quality failures:
     1108Night stack quality failures:
    11091109{{{
    11101110select stack_id,label,data_group,quality,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and quality>0;
    … …  
    11761176
    11771177
    1178 Nightly stack for all filters grizy:
     1178Night stack for all filters grizy -- don't call nightlystack.. but now an additional label/data_group..
    11791179 * label: MD05.GR0
    11801180 * data_group: MD05.GR0.20121104.nightstack.<obsdate>,  MD05.GR0.20121030.nightstack.<obsdate>
    … …  
    11851185----
    11861186= MD06.GR0 =
    1187 ''Status:'' Refstacks and full set nightly stacks finished and distributed. Camera SMFs also distributed.
     1187''Status:'' Refstacks and full set night stacks finished and distributed. Camera SMFs also distributed.
    11881188 * 11/18: all filters all exposures running chip->warp
    11891189 * 11/23: reftest samples running
    … …  
    11911191 * 12/22: repairs mostly finished and restarted refstacks.
    11921192 * 12/31: more corrputions found during distribution, fixed.
    1193  * 01/01/2012: Refstacks and sample nightly stacks finished and distributed.
    1194  * 01/11/12: reprocessing of all nightly stacks possible in V3 started
    1195  * 01/17/12: majority of all nightly stacks on datastore. A few faulted with various issues mainly from stacks with 2-3 inputs on far edge skycells.
     1193 * 01/01/2012: Refstacks and sample night stacks finished and distributed.
     1194 * 01/11/12: reprocessing of all night stacks possible in V3 started
     1195 * 01/17/12: majority of all night stacks on datastore. A few faulted with various issues mainly from stacks with 2-3 inputs on far edge skycells.
    11961196 * 12/21/12: new refstack MD06.refstack.20121221 will replace this one for diffims in the 2012/2013 season
    11971197
    … …  
    12861286 * dist_group: NULL, changed camera to MD06.GR
    12871287
    1288 Reprocessed nightly stack sample under the following label and data_group.
     1288Reprocessed night stack sample under the following label and data_group.
    12891289 * label: MD06.GR0
    12901290 * dist_group: MD06.GR
    1291  * data_group of initial sample of nightly stacks processed with ipp-20111110 tag and using the default 4 or more input warps necessary to make stack
     1291 * data_group of initial sample of night stacks processed with ipp-20111110 tag and using the default 4 or more input warps necessary to make stack
    12921292  * data_group: MD06.GR0.20111122 = y
    12931293  * data_group: MD06.GR0.20111123 = z
    … …  
    12951295  * data_group: MD06.GR0.20111125 = r
    12961296  * data_group: MD06.GR0.20111126 = g
    1297  * data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack (typically the far outer skycells, but as a new case, will need to be looked at in more detail)
     1297 * data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack (typically the far outer skycells, but as a new case, will need to be looked at in more detail)
    12981298  * data_group: MD06.GR0.20120109 = y
    12991299  * data_group: MD06.GR0.20120110 = z
    … …  
    13041304
    13051305Distributed on datastore under
    1306  * ps1-md-GR: sample of nightly stack images only (setting up for full set). SSdiff sample still to be run.
     1306 * ps1-md-GR: sample of night stack images only (setting up for full set). SSdiff sample still to be run.
    13071307 * ps1-md-GR-cat: camera SMFs done (have the data_group condor_MD06.V3_01_dg)
    13081308
    1309 Older nightly stacks are really not that pretty..
     1309Older night stacks are really not that pretty..
    13101310
    13111311Problem/corrupt bundles:
    … …  
    13171317----
    13181318= MD07.GR0 =
    1319 ''Status:'' Refstacks and sample nightly stacks finished and distributed.
     1319''Status:'' Refstacks and sample night stacks finished and distributed.
    13201320 * 10/12: all filters all exposures running chip->warp
    13211321 * 11/6: all warps finished, y-band refstack started.
    13221322 * 11/07: y-band refstack finished, z-band started, reftest samples run
    1323  * 11/08: zirg refstack running, nightly stack samples spanning the period since 4/2009 running.
    1324  * 11/12: full refstack set, nightly stack samples finished and distributed
     1323 * 11/08: zirg refstack running, night stack samples spanning the period since 4/2009 running.
     1324 * 11/12: full refstack set, night stack samples finished and distributed
    13251325 * 6/18/12: staticsky run attempted, see notes below
    13261326 * 01/09/13 [wiki:MD07.refstack.20130102] to replace 2011 season refstack
    … …  
    13281328Coming soon. New observations planned for OC149 (February 7, 2012).
    13291329
    1330 All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.
     1330All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.
    13311331[[Image(MD07_g_smf.png,200px)]] [[Image(MD07ns_z_smf.png,200px)]]
    13321332
    … …  
    14181418 * dist_group: NULL, changed camera to MD07.GR
    14191419
    1420 Reprocessed nightly stack sample under the following label and data_group.
     1420Reprocessed night stack sample under the following label and data_group.
    14211421 * label: MD07.GR0
    14221422 * dist_group: MD07.GR
    1423  * data_group of initial sample of nightly stacks processed with both ipp-20110622 and ipp-20111110 tag (see date) and using the default 4 or more input warps necessary to make stack
     1423 * data_group of initial sample of night stacks processed with both ipp-20110622 and ipp-20111110 tag (see date) and using the default 4 or more input warps necessary to make stack
    14241424  * data_group: MD07.GR0.20111107 = y
    14251425  * data_group: MD07.GR0.20111108 = z
    … …  
    14271427  * data_group: MD07.GR0.20111110 = r
    14281428  * data_group: MD07.GR0.20111111 = g
    1429  * data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack
     1429 * data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack
    14301430  * data_group: MD07.GR0.201201xx = y
    14311431  * data_group: MD07.GR0.201201xx = z
    … …  
    14431443
    14441444Distributed on datastore under
    1445  * ps1-md-GR: sample of nightly stack images only (setting up for full set). SSdiff sample still to be run.
     1445 * ps1-md-GR: sample of night stack images only (setting up for full set). SSdiff sample still to be run.
    14461446 * ps1-md-GR-cat: camera SMFs loading (have the data_group condor_MD07.V3_01_dg)
    14471447
    … …  
    14541454 * 4/18/12: restarted from scratch after raid crash in November/December with the new ipp-20120404 tag to include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map but not the boost factors).
    14551455 * 4/20: gri reprocessing finished and refstacks started. zy prepped but delayed until 4/22 for other tests.
    1456  * 4/21: gri reprocessed camera SMFs and nightly stacks distributed, reference and test stacks also distributed
     1456 * 4/21: gri reprocessed camera SMFs and night stacks distributed, reference and test stacks also distributed
    14571457 * 4/22: z reprocessing started
    14581458 * 4/24: zy reprocessing finished, zy refstack started
    1459  * 4/25: all nightly stack reprocessing finished, all refstacks finished and distributed, staticsky run and distributed.
     1459 * 4/25: all night stack reprocessing finished, all refstacks finished and distributed, staticsky run and distributed.
    14601460
    14611461Coming soon. New observations planned for OC152 (May 5).
    … …  
    15691569
    15701570
    1571 Nightly stacks under the following label and data_group and distributed under ps1-md-GR
     1571Night stacks under the following label and data_group and distributed under ps1-md-GR
    15721572 * label: MD08.GR0
    15731573 * data_group: MD08.GR0.20120417.nightstack.<obsdate> = g
    … …  
    15771577 * data_group: MD08.GR0.20120421.nightstack.<obsdate> = y
    15781578 * dist_group: MD08.GR
    1579  * nightly stack_id/skycell found faulted
     1579 * night stack_id/skycell found faulted
    15801580{{{
    15811581        862445  skycell.087     MD08.GR0        MD08.GR0.20120417.nightstack.20090620   MD08.GR         g.00000 
    … …  
    16121612 * data_group: MD08.refstack.20120422.5x20120425
    16131613 * dist_group: MD.staticsky
    1614  * '''Eric Morganson has noted that the RA_PSF,DEC_PSF is NULL''' http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/ticket/1505, need to track down if fixed and broken again. should be able to use skycell WCS to get coordinates from xy.
     1614 * '''Eric Morganson has noted that the RA_PSF,DEC_PSF is NULL''' http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/ticket/1505, need to track down if fixed and broken again. Should be able to use skycell WCS to get coordinates from xy.
    16151615
    16161616
    … …  
    16651665 * odd images to beware of
    16661666{{{
    1667 -- either tracking jump or mirror solution issue - check if used/rejected in z nightly stack, yes but rejected at end before final stack made by PSF chi2 clip.
     1667-- either tracking jump or mirror solution issue - check if used/rejected in z night stack, yes but rejected at end before final stack made by PSF chi2 clip.
    16681668neb://ipp048.0/gpc1/MD08.GR0.20120420/o5103g0041o.101597/o5103g0041o.101597.ch.442508
    16691669}}}
    … …  
    16761676 * 6/01:  chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71.
    16771677 * 6/02: reftests run for g,y,z
    1678  * 6/03: refstack for g,y started, nightly stacks run.
    1679  * 6/04: refstack for z started with exposures available, full nightly stack processing on hold for z,r,i until solution with XY68,71 found. Will try an do whatever is possible.
    1680  * 6/06: MD09 r,i refstacks started. z refstack, 90% nightly stacks out to datastore.
    1681  * 6/07: refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and nightly stacks that are possible distributed (see listing below for dates of incomplete processing).
    1682  * 7/29 MD09.GR0 XY67,71 chip problem (~48 exposures) cannot be fixed so set to poor quality, remaining camera SMF and nightly stacks processed and distributed.
     1678 * 6/03: refstack for g,y started, night stacks run.
     1679 * 6/04: refstack for z started with exposures available, full night stack processing on hold for z,r,i until solution with XY68,71 found. Will try an do whatever is possible.
     1680 * 6/06: MD09 r,i refstacks started. z refstack, 90% night stacks out to datastore.
     1681 * 6/07: refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and night stacks that are possible distributed (see listing below for dates of incomplete processing).
     1682 * 7/29 MD09.GR0 XY67,71 chip problem (~48 exposures) cannot be fixed so set to poor quality, remaining camera SMF and night stacks processed and distributed.
    16831683 * '''8/31 replaced with MD09.refstack.20120831 to fix the weigh pixel problem causing arc region of lower sensitivity'''
    16841684
    … …  
    16871687
    16881688|| [[Image(MD09_g_smf.png,200px)]] [[Image(MD09ns_z_smf.png,200px)]] ||
    1689 || All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible. ||
     1689|| All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible. ||
    16901690
    16911691
    … …  
    17761776
    17771777Troubled skycells
    1778   * Previously had a few missing/faulted skycells around edges (i-band example left). New refstacks do not have any quality failures so far and note the current nominal position of the populated skycells is shifted leftward (middle) to accomodate the reprocessing of the nightly stacks with the early pointing (right).
     1778  * Previously had a few missing/faulted skycells around edges (i-band example left). New refstacks do not have any quality failures so far and note the current nominal position of the populated skycells is shifted leftward (middle) to accomodate the reprocessing of the night stacks with the early pointing (right).
    17791779
    17801780|| [[Image(MD09.refstack.20100721.NULL.i.00000.stksummap.png,300px)]] || [[Image(MD09.refstack.20120603.NULL.g.00000.stksummap.png,300px)]] ||  [[Image(MD09.GR0.20120602.nightstack.20090629.NULL.g.00000.stksummap.png,300px)]] ||
    … …  
    18081808
    18091809 * sample still unprocessed due to XY71,67 faults. tracing/debugging of new ops tag required to fix.
    1810   * seems to be problem with psphot from ipp-20120531 tag, tests work with ipp-20120626 tag. these will be given a quality fault of 42 so that the exposures and nightly stacks can finish.
     1810  * seems to be problem with psphot from ipp-20120531 tag, tests work with ipp-20120626 tag. these will be given a quality fault of 42 so that the exposures and night stacks can finish.
    18111811{{{
    18121812| chip_id | exp_id | exp_name    | dateobs             | filter  | comment                        |
    … …  
    18691869
    18701870
    1871 __Nightly stacks__: under the following label and data_group and distributed under ps1-md-GR
     1871__Night stacks__: under the following label and data_group and distributed under ps1-md-GR
    18721872 * label: MD09.GR0
    18731873 * data_group: MD09.GR0.20120602.nightstack.<obsdate> = includes all filters
    18741874 * dist_group: MD09.GR
    18751875
    1876  * nightly stack_id/skycell found faulted (5), may be manually recoverable but not in near future.
     1876 * night stack_id/skycell found faulted (5), may be manually recoverable but not in near future.
    18771877{{{
    18781878        Stack ID Skycell ID     Label           data grp                               dist grp         Filter
    … …  
    19051905
    19061906}}}
    1907  * Nigel finds RA_PSF,DEC_PSF still NULL for these -- should be fixed in later versions
     1907 * Nigel finds RA_PSF,DEC_PSF still NULL for these -- should be fixed now in future versions (8/2012)
    19081908
    19091909
    … …  
    19181918
    19191919
     1920Calibrated stack photometry appears suspect to at least 0.03-0.05 (have seen much worse in other fields):
    19201921|| skycell overlap match mag comparison  || [[Image(md09refNx20130128skycal_cmf_i.txt_dmagptmap.png,300px)]] ||
    1921 || refcat match mag comparison: appears 0.03-0.05 mag error in photometry || [[Image(md09refNx20130128skycal_cmf_i.txt_drefmagptmap.png,300px)]] ||
     1922|| refcat match mag comparison example: appears 0.03-0.05 mag error in pixels (staticsky) || [[Image(md09refNx20130128sky_cmf_i.txt_drefmagptmap.png,300px)]] ||
     1923|| refcat match mag comparison example: appears 0.03-0.05 mag error still in photometry recalibration (skycal) || [[Image(md09refNx20130128skycal_cmf_i.txt_drefmagptmap.png,300px)]] ||
    19221924
    19231925