IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 11 and Version 12 of PS1_IPP_Czarlog_20190426


Ignore:
Timestamp:
May 1, 2019, 12:56:31 AM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190426

    v11 v12  
    460460
    461461
     462''' Nightly Science: Featuring Desperate Diff Dan''' by JRF
     463
     464Here we go, desperate diff time boys an girls!
     465
     466It appears that despite a full first chunk, there are some issues.
     467Running the night report, I see:
     468{{{
     469    Current status of SS and Euclid chunks:
     470                 chunk |   Nquad_firstvisit |   Nquad_fourvisits |        Ndiffs done |
     471    OSSR.R10S1.14.Q.w  |                 20 |                 20 |             76/ 80 | chunk not finished
     472    OSSR.R11S2.14.Q.w  |                 20 |                 20 |             80/ 80 |         chunk done
     473}}}
     474(it wasn't quite like this at the time, but close enough). So there are some missing diffs.
     475
     476Doing the following query shows us that indeed there are 2 problem diffs, where the FWHM is very high.
     477{{{
     478mysql -hscidbm -uXXXXXX -pXXXXXX gpc1 -B -e " select exp_id,exp_name,camProcessedExp.path_base,filter,camProcessedExp.fault,fwhm_major,camProcessedExp.quality,camProcessedExp.deteff,sigma_ra,sigma_dec from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where dateobs like '`date -u "+%Y-%m-%d"`%' and quality>0; "
     479
     480exp_id  exp_name        path_base       filter  fault   fwhm_major      quality deteff  sigma_ra        sigma_dec
     4811465386 o8604g0101o     neb://any/gpc1/OSS.nt/2019/05/01//o8604g0101o.1465386/o8604g0101o.1465386.cm.2109315    w.00000   0       15.3435 4007    -4.02871        NULL    NULL
     4821465402 o8604g0117o     neb://any/gpc1/OSS.nt/2019/05/01//o8604g0117o.1465402/o8604g0117o.1465402.cm.2109350    w.00000   0       13.0833 4007    -4.04619        NULL    NULL
     483}}}
     484
     485This is supported by the observers log, https://ps1wiki.ifa.hawaii.edu/trac/wiki/ObsLog20190501 where they mentions issues.
     486
     487The problem chunk is {{{OSSR.R10S1.14.Q.w}}}
     488
     489Find out which visits are the issue:
     490{{{
     491mysql -hscidbm -uXXXXXX -pXXXXXX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R10S1.14.Q.w%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk GROUP BY selchunk.object;"
     492
     493chunk                   object          warp1   warp2   warp3   warp4   workdir                         label                     data_group      reduction
     494OSSR.R10S1.14.Q.w       ps1_30_1462     2084925 2084995 2085050 2085086 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     495OSSR.R10S1.14.Q.w       ps1_30_2554     2084884 2084972 2085007 2085062 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     496OSSR.R10S1.14.Q.w       ps1_30_2555     2084887 2084962 2085012 2085059 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     497OSSR.R10S1.14.Q.w       ps1_30_2711     0       2084924 2085000 2085055 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     498OSSR.R10S1.14.Q.w       ps1_30_2734     2084881 2084956 2085003 2085057 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     499OSSR.R10S1.14.Q.w       ps1_30_2735     2084891 2084970 2085008 2085066 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     500OSSR.R10S1.14.Q.w       ps1_30_2736     2084890 2084965 2085011 2085064 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     501OSSR.R10S1.14.Q.w       ps1_30_2737     2084899 2084969 2085017 2085073 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     502OSSR.R10S1.14.Q.w       ps1_30_2750     2084897 2084979 2085026 2085067 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     503OSSR.R10S1.14.Q.w       ps1_30_2752     2084909 2084990 2085034 2085079 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     504OSSR.R10S1.14.Q.w       ps1_30_2830     2084907 2084980 2085023 2085080 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     505OSSR.R10S1.14.Q.w       ps1_30_2831     2084904 2084988 2085030 2085077 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     506OSSR.R10S1.14.Q.w       ps1_30_2832     2084923 2084992 2085042 2085084 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     507OSSR.R10S1.14.Q.w       ps1_30_2833     0       2084996 2085046 2085082 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     508OSSR.R10S1.14.Q.w       ps1_30_2834     2084902 2084983 2085031 2085078 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     509OSSR.R10S1.14.Q.w       ps1_30_2835     2084917 2084985 2085041 2085083 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     510OSSR.R10S1.14.Q.w       ps1_30_2838     2084912 2084993 2085043 2085081 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     511OSSR.R10S1.14.Q.w       ps1_30_5622     2084920 2085004 2085051 2085085 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     512OSSR.R10S1.14.Q.w       ps1_30_5635     2084908 2084977 2085029 2085074 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     513OSSR.R10S1.14.Q.w       ps1_30_5636     2084901 2084978 2085020 2085070 neb://@HOST@.0/gpc1/20190501    OSS.nightlyscience        OSS.20190501    SWEETSPOT
     514}}}
     515
     516Since visit 1 is the issue, we still need to get our diffs from visit 3 and 4.
     517The diff pairs we need are:
     518{{{
     519    2085000 & 2085055
     520    2085046 & 2085082
     521}}}
     522
     523Take these warp_ids and run the difftool command for creating these diffs:
     524{{{
     525difftool -dbname gpc1 -definewarpwarp -warp_id 2085000 -template_warp_id 2085055 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20190501 -set_reduction SWEETSPOT -simple -rerun -pretend
     526difftool -dbname gpc1 -definewarpwarp -warp_id 2085046 -template_warp_id 2085082 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20190501 -set_reduction SWEETSPOT -simple -rerun -pretend
     527}}}
     528
     529You'll get a response along the lines of:
     530{{{
     531    10:18:46.128181 RINGS.V3 T T 0  0.000000 nan nan nan nan 1
     532}}}
     533
     534But more importantly you should see them in the diffInputSkyfile table:
     535{{{
     536    SELECT * FROM diffInputSkyfile WHERE warp1 = 2085046 AND warp2 = 2085082 LIMIT 100;
     537}}}
     538and here:
     539{{{
     540    http://datastore.ipp.ifa.hawaii.edu/IPP-MOPS-TEST/
     541}}}
     542
     543Bedtime! Will add some more details on Wednesday.
     544
     545
    462546=== Wednesday : 2019.05.01 ===
    463547
    464548TODO:
     549  - Finish my notes for the desperate diffs stuff...
    465550 - configure ippc91 for processing (after it got wiped due to a disk failure). Then put it back into the hosts list
    466551