PSNSC_MOPS requirements and notes (draft) --

Processing products

czar needs to make sure all faults and issues are cleared daily (early morning) for OSS, MSS, ESS, EU products (Bright Euclid.r fields during full moon, ie basically products in quads)

• chip-cam-warp -- nightly
• warp-stack diffims -- nightly
• datastore of WS diffim catalogs -- nightly
• stamps -- chip only from pstamp, MOPS has own stamp server that requires uncompressed image access

• MOPS needs visit 1-2, 2-3 when visit 4 lost due to weather/faults to increase just a pair to triple for tracklets
  • lost visit mapping to be done -- typical losses from fault at camera for any visit due to FWHM cut, v4 lost often due to weather/closure

    v1 missing: v2-v3
    v2 missing: v1-v3
    v3 missing: v2-v4
    v4 missing: v2-v3
    

  • now nightly_science.pl attempts at end-of-night/after observing if DESPERATE_DIFFS true

* sometime seem to be auto-made and other times not? need update on both when known -- appears to be at least partially due to the .multi nightly science task (which is not right and shouldn't be happening as .multi was supposed to be multiple night return visits?) -- .multi look to be made intranight when the camera stage quality faults on visit 2 & 3 and then makes a visit 1-4 diffim since neither 1 nor 4 is used to make a diffim -- expect other combinations to then be possible as well?

• desires first chunk of night to be checked and cleared (desperate/alternate diffs etc) so that NEOs can be identified ASAP -- the desperate/alternate diffs should be done at the end of each chunk if bad camera/diff quality or lack of visit 4 due to weather etc

NEO targeted observations

FSS.nightlyscience, FSS.WS -- also needs to be finished by early morning

• obs_mode FSS (new 201608) -- 6 exposure, MD-like dither (rotation) plus small offsets -- so WSdiff only and published to normal MOPS datastore location
• chip->warp normal stdscience pantasks as with xSS
• WSdiff currently runs in ~ippmops/stdscience_ws to make uncompressed WSdiffs

Cleanup

Specific for MOPS needs

• chip -- 1 days
• warp -- ? days (no specific requirement) -- 2018xxxx keeping all warps on disk now (no cleanup by default)
• WW, WSdiff -- 1 days -- if disk space issue, can reduce this to 2 days as long as warps uncleaned the update time is minimal
  • maybe longer for the FSS WSdiff since special follow-up field
• datastore WSdiff catalog bundles -- Bill has script doing?
• pstamp -- manual cleanup like other ps_ud_%

Tasks/Improvements/Enhancements needed

Nightly products -- only really need image+mask (uncompressed) so could make normal use of no weight file to save disk use during nightly

• currently hacked to not produce mask+weight in ippitc due to overuse of itc datanodes with non-nightly products...
• compression and hack removed ~201709xx once MOPS installed new nodes --

WW diffims for MOPS --

• excess detections causing full rejection of OTA -- typically edges full of junk, unclear if could be improved with other cuts, need real sources on edges as it can still be useful
• also excess detections in poor convolution/noise level that needs to be restricted more

centroids poor causing large GCR -- factor of 2 larger..

• Rob did comparisons of ipp-20141024/ipp-20170121 and products -- appears to be due to issues in all WWdiffs, not problem in SMF and warp, WWdiff, WSdiff image WCS
• will add checks with MOPS tests -- problem in the psphot/centroid code (simple psphotReadoutMinimal probably)
• Serge added GCR+GCD comparison to test reports

astrometry errors --

• chips with bad astrometry getting into warps
• chips with poor astrometry getting into warps
• astrometry in warps large offsets
• astrometry errors seems to have same set of ~5 values --
• ota 17 regular problem -- recent example exposures o8467g0077o, o8467g0095o, o8467g0113o, o8467g0131o

excess detections from poor images w/ telescope jumps/tadpoles/trails -- need check and quality flag in camera stage to prevent from getting to diffs/MOPS

masking -- has been shown useful for pre/recovery having the masked pixels not removed from the images by reprocessing with masking turned off

• previously was label mehtest.nomask, MASK FALSE in hacked CHIP_MASKTEST --

  chiptool -definebyquery -set_label mehtest.nomask  -set_workdir neb://@HOST@.0/meh/mehtest.nomask -set_data_group mehtest.nomask -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name XXXXXX
  

• new command and config to work on this -- also change to make processing more retained since these are images MOPS makes measurements on to submit to MPC

  chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name XXXXX
  

• overall, masking needs to be re-evaluated as appears to be possibly more aggressive than needed and not use properly in others... -- gpc1 and gpc2 both need thorough static (chip) and dynamic (warp) checking -- recent example o8487g0686o(ota75,05),
• masked pixels should be left in the images or in a separate extension and higher level code use use pixels based on mask values -- removal of pixels at start of process prevents any adjusted use of them later in the processing

WS diffims for MOPS --

• image files uncompressed -- either in processing or a copy on ipp032 -- ~10TB/night extra space used (20TB since data kept for 36hrs), processing seems ok (network i/o, dtime, faults were all acceptable) --
• update w-band template stacks -- see below
• could be using WS for i-band always -- need to test detection threshold like done for WWdiff (test chunk is in w-band, but use same threshold for all filters in WWdiff and even that could/should be adjusted at least for dynamic s/n cuts in MOPS) -- i-band LAP stacks not the greatest
• exposure uniqueness required -- update .mops catalog format? (see below)
• publishing -- TBD -- started for PS2 as live comparison 20180xxx --

ppTranslate/ppMOPS -- .mops catalog format update/revision --

• evaluate contents again, remove/add new columns as needed -- must add skycell_id, chip XY position
  • to further reduce neb+gpc# queries, add additional table with all entires and not just best/primary with ID, skycell_id, chip xy, R to skycell center, path on disk (direct, not neb)
• need to re-check if selected detection is best/primary etc
• merge WW and WSdiff catalog publishing to single product if possible -- conflict with timing possibly
• -- likely best to just wait for MOPS to switch to directly using the diff CMF files for both --
• consider java implementation to replace ppMops

w-band stack update for WSdiff --

• best seeing, uniform pixel coverage, WS level depth (about 10-20 best inputs to possibly allow quad stacks as well)
• warps ready, need to run stacking and evaluate coverage (cleanup of poor edge cases)
• by default will use the different LAP.PV3W set RA~7-16hr unless find a problem

z-band 3PI WWdiffs many more detections over i,w (twilight time and now full moon time for Euclid fields) --

MOPS test samples --

• daily MOPS test quad -- label MOPS.dailytestset -- MOPS.dailytestset -- still in development
• ops tag modification test chunk -- MOPS.testchunk.OSSR.R15S2.0.Q.w.20140603
  • necessary comparison set for WW and WS diff for any development/modifications to ops tag
  • need to define a test chunk for i-band probably -- different s/n dynamics? different stacks in WSdiffs at the very least
• updated/recent test chunk 20180412 OSSR.R14S2.14.Q.w -- in development
• special case test quads -- fast movers, comets, other? -- quads need to be selected/defined and run through IPP
• more automated checking for missing visit WWdiffs -- create by exposure status table in czardb for a night reset @initday?
• PS2 test quad --
  • initial early i-band -- OSSR.R13N1.7 (546 known in chunk) night 8252/58251 -- 20180514 -- -- o8252h0040o, o8252h0055o, o8252h0070o, o8252h0085o
  • updated w-band with improved masking (testing) -- OSSR.R20S1.12.Q.w (717 known in chunk) night 8309/58308 -- 20180710 -- o8309h0262o, o8309h0277o, o8309h0292o, o8309h0307o
• PS2 test chunk --

MOPS stamp server @ITC -- MOPS using own node for this now 08/2017 -- node crash 20170917 unclear --

• needs some raid1 disk (minimum few 100 GB, more in future for other work), modest RAM and cores -- more RAM+cores, more stamps can be made
• must be a reliable node and close to the data nodes on the network (and out to neo cluster) -- unclear which is to be used
• will copy off images for stamps -- can be compressed in future -- test ~20170920 if everything ok
• scp/nfs/rcp debate for overheads -- is it possible to use less intensive encryption in ssh?

IPP pstamp server

• backlog -- particularly w/ QUB stamps, in future may want to split pantasks so MOPS doesn't get behind large numbers of uploaded parse jobs
  • can bump up the parse limit (in pstamp.pro) from 10->20 and should help for short term -- added as config in ipp-20170121r40242 update (see below)
• unclear which refcat used in returned stamps (ie for astrometry in MOPS recovery work 2MASS/GAIAv1) -- really useful for warps->diffs to also have but PSREFCAT isn't propagated... -- keyword added now 2018xxxx
• more broken images may be hiding without triggering error in pstamp -- expected pixels seen in other exposure in quad, chip being skipped due to broken?
• broken files is plaguing precovery of priority events (fault 25,26 ) -- reprocessing to fix as needed -- unacceptable that LAP.PV3 files are broken being a recent reprocessing so just jusing DEFAULT recipe and calling mopsfix/mops.fixbrokenLAP.20180323 ; w-band not in LAP.PV3 reprocessing, typically old ecliptic.rp and those not reprocessed for wref stack, so using SWEETSPOT config and calling PSNSC.wfix.20180525
  • ecliptic.rp or LAP.PV3W w-band

    chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name XXXXXX
    

  • LAP.PV3 grizy-band -- probably should have used -set_reduction LAP_SCIENCE

    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name XXXXXX
    

  • once reprocessed, can send the broken file to cleanup to clear from ippMonitor so as not to confuse with other broken files

    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX
    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
    

• errors in updates may happen (red cases on ippMonitor) for ps_ud_MOPS or ps_ud_MOPS.2, setting chip and warp to fully update can check if broken or just in odd fault state for imfile/skyfile(s)

  chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id XXXXXX
  warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
  

• parsing interrupted by auto-restart --

  pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id XXXXX
  pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state goto_cleaned -req_id XXXXX
  

Ops tag notes

ipp-20170121 -- r40242 starting 20171205, cam_id>=1991047 is PSREFCAT= 'PS1.REF.20170919' -- key change/update refcat using GAIAv1 astrometry and CMF_DV5 format

• new refcat for astronomy with GAIAv1 but photometry still PV3 ubercal (PSREFCAT PS1.REF.20170919, catdir.refcat.20170919.v0)
• cleared bugs to update difference catalog CMF to DV5 version (additional columns in CMF but none added to ppmops file), will allow for adding any new diff params in future as needed
• adjusted config allowing for more detections along edges to more closely match the prior tag (PSF_FIT_MODEL_SUM_FRAC_CUT,PSF_FIT_MASKED_SUM_FRAC_CUT)
• pstamp config options to change parse and number of faults limits to reduce backlog in getting stamps
• various bug fixes that were stuck in the xxxxx-stack diff processing when processing changed to the ipp-20170121
• WSdiff configuration for a lower signal-to-noise cut similar to WWdiffs (SWEETSPOT_WS)

ipp-20170121 -- rX starting 201702xx -- key changes/updates on WCS but also many modifications throughout code for PV3 (with bugs for nightly processing..)

• problem in WCS changes between tags and updates of pixels -- night of 20170214 issue?
• full use of tag started on

ipp-20141024 -- rX starting -- setup for post-ps1sc processing, many revisions

• using refcat PSREFCAT PS1.REF.20140713, catdir.refcat.20140713.v0