PSST QUB requirements and notes (draft) --

Processing products

Like for MOPS, czar needs to make sure all faults are cleared daily for OSS, MSS, ESS, ThreePi products

• chip-cam-warp -- nightly
• warp-stack diffims -- nightly
• datastore of WS diffim catalogs -- nightly
• PSS and rapid updates for historical stamps (warp+stack+diff) --

QUB targeted observations (QUB.nightlyscience, QUB.WS/SS) should only be managed by Mark Huber -- notes on psweb.mp.qub.ac.uk/psat-lv-wiki/index.php/Pan-STARRS_pages

Processing timescale

Ideal and last cutoff times for diffim catalogs to be available for QUB ingestion and processing (TBD) --

• chip-cam-warp -- nightly
• warp-stack diffims -- nightly

Cleanup

(not up to date 20180124)

• chip -- 2 days (same as MOPS)
• warp -- 4 days (so weekends covered)
• WSdiff -- 4 days (so weekends covered) -- if disk space issue, can reduce this to 2 days as long as warps uncleaned the update time is minimal
• datastore WSdiff catalog bundles -- 1 month (manually done)
• ps_ud_QUB -- manual, only Mark Huber should deal with
• QUB.nightlyscience, QUB.WS/SS -- manual, only Mark Huber should deal with

Tasks to do

(not up to date 20180124) in order of possible accomplishment --

• only really need image+mask (compressed so ~ok) -- like MOPS could switch off weight image for less disk use in processing
• gpc2 import of PV3 stack products -- requires new ops tag and then export/import of gpc1 stack rows to gpc2
• WSdiff daily test set --
• PV3 w-band stacks -- some done at LANL, likely a good amount of reorganization work to be done as well as compute and disk space required -- some exist as of 20160201, have asked QUB if they wish to partially switch over but suspect not with all the other work going on

• gri synthetic w-band stacks for all sky -- much development work, nodes and disk space required.

• PSS should have option to turn up power for updates when large number of historical stamps are required (method to do so is unclear/undocumented..)
• cleanup timescales -- 3PI(7d)+OSS(2d)+SNIa(?) are all various times..
• gpc1 on ipp0012 restored for access to what diffims should be available --

• switch to PV3 stacks for grizy (done 201xxxx) -- requires new ops tag, still some bugs in it -- ~ippqub running modified old tag to use 20141024 since ipp-20170121 broken for WSdiff

• s/n cut in diffphot -- MOPS will use WSdiff in future at lower cut ~3.5-4, will the be useful for QUB (or at least able to filter to their desired s/n cut?)

• good_frac limit 0.2 reduce to 0 (remove constraint) to match MOPS WWdiff if possible

• gpc1 replication to ipp003 (tcs mysql) for QUB+ use -- will be retiring machines soon so need to work in that time frame

Processing stage time summary

Based on 20180122, 20180123 --

• nightly rates limited by summitcopy, ch-cam-wrp ~60 exp/hr so keeps up with observations (generally can catchup to some degree ~80exp/hr possible?), WSdiff+distribution can match ~60-80 exp/hr without much modification
• dtime_script -- Nsources dependent generally, but on average (seconds) | sigma | min | max |
  • ch -- 118.48012954361 | 34.706445650573 | 51.000003814697 | 668 |
  • cam -- 448.97415129087 | 102.86848443046 | 9.9999856948853 | 1001 |
  • wrp -- 63.084047381327 | 30.766145179551 | 12.999969482422 | 680 |
  • WSdiff -- 102.62499931014 | 31.209872126263 | 56.000019073486 | 672 |
• timestamps using dateobs, diffRun.registered, distRun.time_stamp disappointing ~30-50 minutes for exposure through WSdiff to make it to datastore

  | exp_name    | exp_time | dateobs             | registered          | time_stamp          | chip_id | cam_id  | warp_id | diff_id | dist_id |
  +-------------+----------+---------------------+---------------------+---------------------+---------+---------+---------+---------+---------+
  | o8141g0062o |      120 | 2018-01-23 05:13:03 | 2018-01-23 05:40:40 | 2018-01-23 06:04:29 | 2048378 | 2014852 | 1991114 | 1651871 | 5171373 | 
  | o8141g0187o |       45 | 2018-01-23 07:12:22 | 2018-01-23 07:40:39 | 2018-01-23 08:08:56 | 2048501 | 2014975 | 1991234 | 1651997 | 5171444 | 
  | o8141g0249o |       45 | 2018-01-23 08:13:04 | 2018-01-23 08:40:38 | 2018-01-23 09:21:44 | 2048561 | 2015036 | 1991299 | 1652074 | 5171500 |
  | o8141g0440o |       45 | 2018-01-23 11:11:03 | 2018-01-23 11:35:39 | 2018-01-23 11:43:22 | 2048720 | 2015195 | 1991452 | 1652210 | 5171550 |  
  | o8141g0445o |       45 | 2018-01-23 11:15:53 | 2018-01-23 11:40:39 | 2018-01-23 11:57:05 | 2048725 | 2015200 | 1991459 | 1652216 | 5171556 | 
  | o8141g0448o |      120 | 2018-01-23 16:02:14 | 2018-01-23 16:25:41 | 2018-01-23 16:34:55 | 2048727 | 2015202 | 1991463 | 1652218 | 5171558 | 
  
  

  • manually crontab queuing WSdiffs still -- sped up somewhat -- need full night to see
• pstamp and updates can be limiting factor as well for flagging real transients --
  • need some nights of good weather and requests to check on
  • need to adjust warp+WSdiff pixel cleanup to match QUB request window -- suspect 1 week for warp and 3 day for WSdiff would be sufficient (current 2/60hr day warp, 1 day/36hr WSdiff)
  • splitting out QUB updates to dedicated nodes also useful -- depends on timing of the submissions and nodes to be allocated