Changes between Version 65 and Version 66 of STS.refstack.20120510
- Timestamp:
- Jun 21, 2012, 12:39:29 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
STS.refstack.20120510
v65 v66 785 785 Nights to pick for test -- do test run skycell.1.03 again since edge field and can have overlap. Not clear way to get FWHM other than from gpc1 DB, need to know what chips go into that? No, camProcessedExp sufficient. 786 786 * need to know rough FWHM of refstack and nightly exposures to pick sample -- rough FWHM for refstack in log of run, nightly exposures in DB 787 * refstack ~3.94 787 * refstack ~3.94 unconv, ~5.5 conv -- this could be a problem, many warps << than this so stack will be deconvolved 788 788 {{{ 789 789 SELECT exp_id,exp_time,dateobs,zpt_obs,zpt_stdev,fwhm_major,format(fwhm_major/fwhm_minor,3) as elong,format(iq_fwhm_major/iq_fwhm_minor,3) as iq_elong,format(fwhm_major-iq_fwhm_major,2) as fwhm_iq_maj,format(camProcessedExp.bg,2) as bg,format(camProcessedExp.bg_stdev,2) as bg_stdev,format(fwhm_major_uq,2) as uq,format(fwhm_major_lq,2) as lq,n_stars,comment FROM rawExp join chipRun using (exp_id) join camRun using (chip_id) join camProcessedExp using (cam_id) WHERE comment like "STS%PP1%" group by exp_id; … … 791 791 * 120420 - look ok 792 792 * 120507 ~7.52 (7.9 in cam, wrong way even for cam/warp scaling) -- looked poor, don't use as test 793 794 793 * need to pick one from 2011 not used in refstack 795 794 {{{ … … 799 798 }}} 800 799 800 801 801 2012 sample to test before full field updating -- like from above 802 802 {{{ … … 822 822 }}} 823 823 824 * o6054g0305o on disk still so easy to just test try also825 * try to at least include 4/14 using a GIQ? 4/20 cleaned but know work , so including 4/14 not necessary.826 * have to use the conv stack 824 * o6054g0305o on disk still, so easy to just test try also even if not great 825 * try to at least include 4/14 using a GIQ? 4/20 cleaned but know works, so including 4/14 not necessary. 826 827 827 828 828 Run test diffims: 829 * currently have to use the conv stack, adding c to script/files 829 830 {{{ 830 831 select uri from warpSkyfile where warp_id=409660 and skycell_id like "%1.03"; … … 857 858 }}} 858 859 859 * remaining set 860 {{{ 860 {{{ 861 -- remaining set 862 861 863 ./run_ppsub_night_ws.csh gpc1/STS.nt/2012/04/23//o6040g0441o.479810/o6040g0441o.479810.wrp.401931.skycell.1.03.fits gpc1/STS.reftest.20120510/STS.V4/skycell.1.03/STS.V4.skycell.1.03.stk.899918.fits 20120423c > runppsub_sts103_120423_1c.bat 862 864 source runppsub_sts103_120423_1c.bat >& runppsub_sts103_120423_1c.log & … … 877 879 878 880 * looks like 20110918, 20120420, 20120423/20120502 are good samples -- use 4/23, seeing some odd conv/pixel rejected artifacts(?) coming out in diffim but not sure if detected and 30s version 879 * may also want to try noconv since STS is supposed to be done within certain IQ constraints880 881 882 Update full field for diffim run:881 * may also want to try noconv diffims since STS is supposed to be done within certain IQ constraints 882 883 884 Update full field examples for diffim run: 883 885 * set selected nights up for updates - number of exposures ~16(20s)/16(20s)/12(30s) for 20110918, 20120420, 20120423 respectively 884 * label STS.mehdifftest 885 * will need to script mysql call based on what to get the chip/warp_id? detrend different since? may not be unique run given reprocessing for refstack oh well.. need PP1 as well 886 * manually checked and all unique.. 886 * label = STS.mehdifftest 887 * will need to script mysql call based on what to get the chip/warp_id. 888 * detrend different since? may not be unique run given reprocessing for refstack oh well.. manually checked and all unique.. 889 * 20110918 during magicked era, keep eye out for any issues. may have to just reprocess it - try update since just a test anyways and see. possibly have some clean state issues, so keep eye on -- did update for test case anyways so should work 887 890 {{{ 888 891 select exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" group by exp_name; … … 891 894 warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id 892 895 }}} 893 894 * will have some clean state issues possibly.. 895 * note dateobs is 2 groups.. 896 897 * 20110918 during magicked era, keep eye out for any issues. may have to just reprocess it - try update since just a test anyways and see -- did update for test case anyways so should work 898 * create chip->warp update lists 896 897 * create specific chip->warp update lists 899 898 {{{ 900 899 mysql -hippdb01 -uippuser -pxxxx gpc1 -B -e 'select distinct exp_name,dateobs,chip_id,warp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where comment like "STS%PP1%" and dateobs like "2011-09-18%" and warpRun.state like "cleaned" and chipRun.state like "cleaned" group by exp_name; ' | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label STS.mopsdiff.mehtest -chip_id "$4"; warptool -dbname gpc1 -setskyfiletoupdate -set_label STS.mopsdiff.mehtest -warp_id "$5}' > ~/updatechipwarp_sts.mopsdiff.mehtest_110918.bat … … 905 904 }}} 906 905 907 * warp updates driving ipp017 nuts.. ~10-20% RAM for 5 runs for total ~75%, cpu_wait and swap overload. killing 1 at time...906 * warp updates driving ipp017 nuts.. ~10-20% RAM for 5 runs for total ~75%, cpu_wait and swap overload. killing off 1 at time... 908 907 {{{ 909 908 1 ipp017 BUSY 2993.25 0.0.5.fe34 0 warp_skycell.pl --threads @MAX_THREADS@ --warp_id 400730 --warp_skyfile_id 34217254 --skycell_id skycell.1.22 --tess_dir STS.V4 --camera GPC1 --outroot neb://ipp017.0/gpc1/STS.nt/2012/04/20//o6037g0429o.478506/o6037g0429o.478506.wrp.400730.skycell.1.22 --redirect-output --run-state update --reduction STS_DATASET --dbname gpc1 --verbose … … 914 913 }}} 915 914 916 * hm, didn't fault on czarpage for revert? fault=4915 * odd, didn't fault on czarpage for revert? 917 916 {{{ 918 917 select warpRun.warp_id,state,data_state,fault from warpRun join warpSkyfile using (warp_id) where label like "STS.mopsdiff.mehtest" and data_state="update"; … … 927 926 * data_group = STS.mopsdiff.mehtest 928 927 * workdir = meh/STS.mopsdiff.mehtest/ 929 * distribution? no, need to know about publishing - does it need normal distribution? should run on own using diffim info 930 {{{ 931 --> difftool command for normal WS? 932 --> need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied 933 --> what about -exposure? would also seem want -- not part of definewarpstack, only definerun since implied 934 --> also -input_label becomes -warp_label, -template_label becomes stack_label 935 --> need -pretend to see output -- cant test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.% 936 *** wildcards not work so need to do mysql cmdline batch since didnt restrict batch updates.. or who cares, just run them.. this is a problem with the large overlap skycells, and more restrictions will be needed in future runs 937 }}} 928 * dist_group? no, but need to know about publishing - should run on own using diffim info 929 930 * difftool command for normal WS? -definewarpstack 931 * need to add inverse since WS has as false, might as well set diff_mode as well -- not part of definewarpstack since implied 932 * what about -exposure? would also seem want -- not part of definewarpstack, only definerun since implied 933 * also -input_label becomes -warp_label, -template_label becomes stack_label 934 * need -pretend to see output -- can't test until updates are done -- but can intermediate test what to expect, see diffs with other fields so need -skycell_id %.1.% 935 * wildcards not work so need to do mysql cmdline batch since didn't restrict batch updates.. this is a problem with the large overlap skycells, more restrictions will be needed in future runs 938 936 939 937 {{{ … … 945 943 difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -bothways -set_reduction WARPSTACK -skycell_id skycell.1.03 -simple 946 944 947 *** oops, by running 1 skycell, now thinks is down.. forgot.. now need to use -rerun and guess cleanup the other one -945 --> oops, by running 1 skycell, now thinks is done.. now need to use -rerun and guess cleanup the other one - 948 946 difftool -updaterun -set_state goto_cleaned -set_label goto_cleaned -label STS.mopsdiff.mehtest -dbname gpc1 949 947 950 --> huh.. didnt set data_group right, used the base one STS.date.. wonder why..948 --> also seems to have set data_group to STS.<date> and not used the -set_data_group, why? 951 949 952 950 difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label STS.mopsdiff.mehtest -stack_label STS.refstack.20120510 -set_dist_group NULL -set_label STS.mopsdiff.mehtest -set_workdir neb://@HOST@.0/meh/STS.mopsdiff.mehtest -set_data_group STS.mopsdiff.mehtest -set_reduction WARPSTACK -bothways -simple -rerun 953 951 }}} 954 952 955 ''' trouble -- left -bothways in run and don't want to do that'''956 957 * odd fault 4, won't fix-- something in psphot953 '''-- mistake -- also left -bothways in diffim run and don't want to do that''' but finished okay. 954 955 * odd fault 4, revert not fixing -- something in psphot 958 956 {{{ 959 957 242183 skycell.1.29 STS.mopsdiff.mehtest STS.20110918
