Changes between Version 13 and Version 14 of PSST_QUB

Timestamp:

Jan 24, 2018, 3:06:37 PM (9 years ago)

Author:

Mark Huber

Comment:

--

PSST_QUB

@@ -1 +1 @@
 PSST QUB requirements and notes (draft) --
 
+[[PageOutline]]
 
-Processing products -- like for MOPS, czar needs to make sure all faults are cleared daily for OSS, MSS, ESS, ThreePi products
+== Processing products == 
+Like for MOPS, czar needs to make sure all faults are cleared daily for OSS, MSS, ESS, ThreePi products
  * chip-cam-warp -- nightly
  * warp-stack diffims -- nightly
@@ -9 +11 @@
 
 
-QUB targeted observations (QUB.nightlyscience, QUB.WS/SS) should only be managed by Mark Huber.
+QUB targeted observations (QUB.nightlyscience, QUB.WS/SS) should only be managed by Mark Huber -- notes on psweb.mp.qub.ac.uk/psat-lv-wiki/index.php/Pan-STARRS_pages
 
-Processing timescale -- ideal and last cutoff times for diffim catalogs to be available for QUB ingestion and processing (TBD) 
+
+== Processing timescale == 
+Ideal and last cutoff times for diffim catalogs to be available for QUB ingestion and processing (TBD) -- 
  * chip-cam-warp -- nightly
  * warp-stack diffims -- nightly 
 
 
-Cleanup -- 
+== Cleanup == 
+(not up to date 20180124)
  * chip -- 2 days (same as MOPS)
  * warp -- 4 days (so weekends covered)
  * WSdiff -- 4 days (so weekends covered) -- if disk space issue, can reduce this to 2 days as long as warps uncleaned the update time is minimal
  * datastore WSdiff catalog bundles --  1 month (manually done)
- * ps_ud_QUB -- manual, only Mark Huber should do
- * QUB.nightlyscience, QUB.WS/SS -- manual, only Mark Huber should do
+ * ps_ud_QUB -- manual, only Mark Huber should deal with
+ * QUB.nightlyscience, QUB.WS/SS -- manual, only Mark Huber should deal with
 
  
-Tasks to do -- in order of possible accomplishment
+== Tasks to do == 
+(not up to date 20180124)
+in order of possible accomplishment -- 
  * only really need image+mask (compressed so ~ok) -- like MOPS could switch off weight image for less disk use in processing 
  * gpc2 import of PV3 stack products -- requires new ops tag and then export/import of gpc1 stack rows to gpc2
@@ -48 +55 @@
 
  * gpc1 replication to ipp003 (tcs mysql) for QUB+ use -- will be retiring machines soon so need to work in that time frame
+
+
+== Processing stage time summary ==
+Based on 20180122, 20180123 --
+ * nightly rates limited by summitcopy, ch-cam-wrp ~60 exp/hr so keeps up with observations (generally can catchup to some degree ~80exp/hr possible?), WSdiff+distribution can match ~60-80 exp/hr without much modification 
+ * dtime_script -- Nsources dependent generally, but on average (seconds) | sigma | min | max |
+  * ch -- 118.48012954361 |      34.706445650573 |   51.000003814697 |               668 |  
+  * cam -- 448.97415129087 |      102.86848443046 |   9.9999856948853 |              1001 |
+  * wrp -- 63.084047381327 |      30.766145179551 |   12.999969482422 |               680 |
+  * WSdiff -- 102.62499931014 |      31.209872126263 |   56.000019073486 |               672 | 
+ * timestamps using dateobs, diffRun.registered, distRun.time_stamp disappointing ~30-50 minutes for exposure through WSdiff to make it to datastore 
+{{{
+| exp_name    | exp_time | dateobs             | registered          | time_stamp          | chip_id | cam_id  | warp_id | diff_id | dist_id |
++-------------+----------+---------------------+---------------------+---------------------+---------+---------+---------+---------+---------+
+| o8141g0062o |      120 | 2018-01-23 05:13:03 | 2018-01-23 05:40:40 | 2018-01-23 06:04:29 | 2048378 | 2014852 | 1991114 | 1651871 | 5171373 | 
+| o8141g0448o |      120 | 2018-01-23 16:02:14 | 2018-01-23 16:25:41 | 2018-01-23 16:34:55 | 2048727 | 2015202 | 1991463 | 1652218 | 5171558 | 
+| o8141g0249o |       45 | 2018-01-23 08:13:04 | 2018-01-23 08:40:38 | 2018-01-23 09:21:44 | 2048561 | 2015036 | 1991299 | 1652074 | 5171500 |
+| o8141g0440o |       45 | 2018-01-23 11:11:03 | 2018-01-23 11:35:39 | 2018-01-23 11:43:22 | 2048720 | 2015195 | 1991452 | 1652210 | 5171550 |  
+| o8141g0445o |       45 | 2018-01-23 11:15:53 | 2018-01-23 11:40:39 | 2018-01-23 11:57:05 | 2048725 | 2015200 | 1991459 | 1652216 | 5171556 | 
+}}}
+  * manually crontab queuing WSdiffs still -- sped up somewhat -- need full night to see
+ * pstamp and updates can be limiting factor as well for flagging real transients --  
+  * need to adjust warp+WSdiff pixel cleanup to match QUB request window -- suspect 1 week for warp and 3 day for WSdiff would be sufficient (current 2/60hr day warp, 1 day/36hr WSdiff) 
+  * splitting out QUB updates to dedicated nodes also useful -- depends on timing and nodes to be allocated
+
+
+
+
+