| | 1 | = simtest FAQ = |
| | 2 | |
| | 3 | == Prerequisites for running a basic simtest are: == |
| | 4 | * building and installing ipp software |
| | 5 | * installing configuration files |
| | 6 | |
| | 7 | == simtest steps: == |
| | 8 | === Create a database (ipp001 is the mysql server on the manoa cluster): === |
| | 9 | {{{ |
| | 10 | % mysql -h ipp001 -u ipp -p |
| | 11 | Enter password: |
| | 12 | mysql> create database dbname; |
| | 13 | Query OK, 1 row affected (0.00 sec) |
| | 14 | }}} |
| | 15 | |
| | 16 | === Change to a working directory (preferably empty) === |
| | 17 | |
| | 18 | === Start a pantasks server === |
| | 19 | {{{ |
| | 20 | %pantasks |
| | 21 | Welcome to pantasks - parallel task scheduler |
| | 22 | |
| | 23 | |
| | 24 | pantasks: |
| | 25 | }}} |
| | 26 | |
| | 27 | === Load the simtest configuration: === |
| | 28 | {{{ |
| | 29 | pantasks: module simtest.pro |
| | 30 | }}} |
| | 31 | |
| | 32 | === Start a simtest === |
| | 33 | |
| | 34 | pantasks: simtest dbname hostname init |
| | 35 | }}} |
| | 36 | |
| | 37 | dbname is the database name, host is the host on which you're running the simtest, and runlevel is either new, inject, or run. |
| | 38 | |
| | 39 | If you are using a newly created database you want to use the "new" option. You can get more information through pantasks: |
| | 40 | {{{ |
| | 41 | pantasks: simtest |
| | 42 | USAGE: simtest (dbname) (hostname) (init) |
| | 43 | if (init) == run : just run the analysis tasks |
| | 44 | if (init) == inject : (re)create the database and (re)inject the images |
| | 45 | if (init) == new : restart completely from scratch |
| | 46 | }}} |
| | 47 | |
| | 48 | You're asked if you want to delete the tables in your database: |
| | 49 | {{{ |
| | 50 | *** delete the tables from database dbname? *** |
| | 51 | *** to delete the tables, answer YES, and give password *** |
| | 52 | *** WARNING: this action is permanent *** |
| | 53 | |
| | 54 | delete the tables (YES/[n]): YES |
| | 55 | enter dbh connection password: |
| | 56 | }}} |
| | 57 | |
| | 58 | Simulated data is created an injected into the database: |
| | 59 | {{{ |
| | 60 | 2009-05-19 02:26:38Z|ipp022 |I|ppSim |
| | 61 | Adding 0 stars between 15.500000 and 20.727417 mag |
| | 62 | 2009-05-19 02:26:42Z|ipp022 |4|psastro |
| | 63 | no supplied reference header data |
| | 64 | output catalog has 7234 stars (36170 measures, 5 secfilt) |
| | 65 | SUCCESS |
| | 66 | 2009-05-19 02:26:45Z|ipp022 |I|psastro |
| | 67 | ran getstar : 3.141629 sec |
| | 68 | 2009-05-19 02:26:46Z|ipp022 |I|psastro |
| | 69 | loaded 7234 reference stars : 0.239388 sec |
| | 70 | 2009-05-19 02:26:46Z|ipp022 |I|ppSim |
| | 71 | Adding 7234 reference stars |
| | 72 | |
| | 73 | Running [/data/ipp003.0/ippTests/psconfig/default.lin64/bin/pxinject -newImfile |
| | 74 | -exp_id 36 -tmp_class_id file.0 -uri file://data/ipp003.0/ippTests/simtest_dir/ |
| | 75 | raw/simtest.001.015.fits -dbname dbname]... |
| | 76 | Running [/data/ipp003.0/ippTests/psconfig/default.lin64/bin/pxinject - |
| | 77 | updatenewExp -exp_id 36 -state run -dbname dbname]... |
| | 78 | exp_name : simtest.001.016. |
| | 79 | 1 files in fileset. |
| | 80 | }}} |
| | 81 | |
| | 82 | After the images are injected you get a pantasks prompt. You can get the status of your simtest with the "status" command: |
| | 83 | {{{ |
| | 84 | pantasks: status |
| | 85 | |
| | 86 | Scheduler is running |
| | 87 | Controller is running |
| | 88 | |
| | 89 | Task Status |
| | 90 | AV Name Njobs Ngood Nfail Ntime Command |
| | 91 | ++ automate.block 5 5 0 0 dettool |
| | 92 | ++ automate.check 118 118 0 0 detselect |
| | 93 | ++ automate.launch 1 1 0 0 dettool |
| | 94 | ++ automate.regular 120 120 0 0 stacktool |
| | 95 | ++ register.imfile.load 125 125 0 0 regtool |
| | 96 | ++ register.imfile.run 76 76 0 0 register_imfile.pl |
| | 97 | ++ register.exp.load 119 119 0 0 regtool |
| | 98 | ++ register.exp.run 76 76 0 0 register_exp.pl |
| | 99 | ++ detrend.process.load 120 120 0 0 dettool |
| | 100 | ++ detrend.process.run 20 20 0 0 detrend_process_imfile.pl |
| | 101 | ++ detrend.processexp.load 120 120 0 0 dettool |
| | 102 | ++ detrend.processexp.run 20 18 0 0 detrend_process_exp.pl |
| | 103 | ++ detrend.stack.load 120 119 0 0 dettool |
| | 104 | ++ detrend.stack.run 1 0 0 0 detrend_stack.pl |
| | 105 | ++ detrend.normstat.load 116 116 0 0 dettool |
| | 106 | ++ detrend.normstat.run 0 0 0 0 (dynamic) |
| | 107 | ++ detrend.norm.load 125 124 0 0 dettool |
| | 108 | ++ detrend.norm.run 0 0 0 0 (dynamic) |
| | 109 | ++ detrend.normexp.load 119 119 0 0 dettool |
| | 110 | ++ detrend.normexp.run 0 0 0 0 (dynamic) |
| | 111 | ++ detrend.resid.load 122 122 0 0 dettool |
| | 112 | ++ detrend.resid.run 0 0 0 0 (dynamic) |
| | 113 | ++ detrend.residexp.load 119 119 0 0 dettool |
| | 114 | ++ detrend.residexp.run 0 0 0 0 (dynamic) |
| | 115 | ++ detrend.reject.load 121 120 0 0 dettool |
| | 116 | ++ detrend.reject.run 0 0 0 0 (dynamic) |
| | 117 | ++ detrend.correct.load 122 121 0 0 dettool |
| | 118 | ++ detrend.correct.run 0 0 0 0 (dynamic) |
| | 119 | ++ flatcorr.addcamera 18 18 0 0 flatcorr |
| | 120 | ++ flatcorr.load 119 119 0 0 flatcorr |
| | 121 | ++ flatcorr.run 0 0 0 0 (dynamic) |
| | 122 | ++ chip.imfile.load 123 123 0 0 chiptool |
| | 123 | ++ chip.imfile.run 0 0 0 0 (dynamic) |
| | 124 | ++ chip.advanceexp 17 17 0 0 chiptool |
| | 125 | -+ chip.cleanup.load 0 0 0 0 (dynamic) |
| | 126 | -+ chip.cleanup.run 0 0 0 0 (dynamic) |
| | 127 | ++ camera.exp.load 120 119 0 0 camtool |
| | 128 | ++ camera.exp.run 0 0 0 0 (dynamic) |
| | 129 | -+ camera.cleanup.load 0 0 0 0 (dynamic) |
| | 130 | -+ camera.cleanup.run 0 0 0 0 (dynamic) |
| | 131 | ++ fake.imfile.load 125 125 0 0 faketool |
| | 132 | ++ fake.imfile.run 0 0 0 0 (dynamic) |
| | 133 | ++ fake.advanceexp 17 17 0 0 faketool |
| | 134 | -+ fake.cleanup.load 0 0 0 0 (dynamic) |
| | 135 | -+ fake.cleanup.run 0 0 0 0 (dynamic) |
| | 136 | ++ warp.exp.load 120 120 0 0 warptool |
| | 137 | ++ warp.exp.run 0 0 0 0 (dynamic) |
| | 138 | ++ warp.skycell.load 120 120 0 0 warptool |
| | 139 | ++ warp.skycell.run 0 0 0 0 (dynamic) |
| | 140 | ++ warp.advancerun 17 17 0 0 warptool |
| | 141 | -+ warp.cleanup.load 0 0 0 0 (dynamic) |
| | 142 | -+ warp.cleanup.run 0 0 0 0 (dynamic) |
| | 143 | ++ diff.skycell.load 121 121 0 0 difftool |
| | 144 | ++ diff.skycell.run 0 0 0 0 (dynamic) |
| | 145 | -+ diff.cleanup.load 0 0 0 0 (dynamic) |
| | 146 | -+ diff.cleanup.run 0 0 0 0 (dynamic) |
| | 147 | ++ stack.skycell.load 121 121 0 0 stacktool |
| | 148 | ++ stack.skycell.run 0 0 0 0 (dynamic) |
| | 149 | -+ stack.cleanup.load 0 0 0 0 (dynamic) |
| | 150 | -+ stack.cleanup.run 0 0 0 0 (dynamic) |
| | 151 | |
| | 152 | Jobs in Pantasks Queue |
| | 153 | 8: detrend.processexp.run 2591 detrend_process_exp.pl (6aaec0) |
| | 154 | 8: detrend.stack.run 2626 detrend_stack.pl (6be410) |
| | 155 | 8: detrend.processexp.run 2634 detrend_process_exp.pl (697ef0) |
| | 156 | 8: detrend.correct.load 2919 dettool (68bc10) |
| | 157 | 8: detrend.norm.load 2920 dettool (67c210) |
| | 158 | 8: detrend.reject.load 2921 dettool (6a0e40) |
| | 159 | 8: detrend.stack.load 2922 dettool (688540) |
| | 160 | 8: camera.exp.load 2923 camtool (68bb80) |
| | 161 | }}} |
| | 162 | |
| | 163 | To speed up your simtest you can add additional pclients with the command: |
| | 164 | pantasks: controller host add hostname |
| | 165 | HostID: 2 |
| | 166 | |
| | 167 | However, there are some caveats. First, adding addional hosts is generally not permitted on nodes in the MHPCC cluster. Second, the Manoa development machines have four cores per node so adding three hosts will monopolize all resources. |
| | 168 | |
| | 169 | === Status output for a successful simtest: === |
| | 170 | {{{ |
| | 171 | pantasks: status |
| | 172 | |
| | 173 | Scheduler is running |
| | 174 | Controller is running |
| | 175 | |
| | 176 | Task Status |
| | 177 | AV Name Njobs Ngood Nfail Ntime Command |
| | 178 | ++ automate.block 5 5 0 0 dettool |
| | 179 | ++ automate.check 593 593 0 0 detselect |
| | 180 | ++ automate.launch 7 7 0 0 chiptool |
| | 181 | ++ automate.regular 1095 1095 0 0 difftool |
| | 182 | ++ register.imfile.load 1094 1094 0 0 regtool |
| | 183 | ++ register.imfile.run 76 76 0 0 register_imfile.pl |
| | 184 | ++ register.exp.load 1093 1093 0 0 regtool |
| | 185 | ++ register.exp.run 76 76 0 0 register_exp.pl |
| | 186 | ++ detrend.process.load 1098 1097 0 0 dettool |
| | 187 | ++ detrend.process.run 75 75 0 0 detrend_process_imfile.pl |
| | 188 | ++ detrend.processexp.load 1094 1093 0 0 dettool |
| | 189 | ++ detrend.processexp.run 75 75 0 0 detrend_process_exp.pl |
| | 190 | ++ detrend.stack.load 1092 1092 0 0 dettool |
| | 191 | ++ detrend.stack.run 5 5 0 0 detrend_stack.pl |
| | 192 | ++ detrend.normstat.load 1089 1089 0 0 dettool |
| | 193 | ++ detrend.normstat.run 5 5 0 0 detrend_norm_calc.pl |
| | 194 | ++ detrend.norm.load 1106 1106 0 0 dettool |
| | 195 | ++ detrend.norm.run 5 5 0 0 detrend_norm_apply.pl |
| | 196 | ++ detrend.normexp.load 1091 1090 0 0 dettool |
| | 197 | ++ detrend.normexp.run 5 5 0 0 detrend_norm_exp.pl |
| | 198 | ++ detrend.resid.load 1085 1085 0 0 dettool |
| | 199 | ++ detrend.resid.run 75 75 0 0 detrend_resid_imfile.pl |
| | 200 | ++ detrend.residexp.load 1093 1093 0 0 dettool |
| | 201 | ++ detrend.residexp.run 75 75 0 0 detrend_resid_exp.pl |
| | 202 | ++ detrend.reject.load 1097 1096 0 0 dettool |
| | 203 | ++ detrend.reject.run 7 7 0 0 detrend_reject_exp.pl |
| | 204 | ++ detrend.correct.load 1098 1098 0 0 dettool |
| | 205 | ++ detrend.correct.run 0 0 0 0 (dynamic) |
| | 206 | ++ flatcorr.addcamera 168 168 0 0 flatcorr |
| | 207 | ++ flatcorr.load 1093 1093 0 0 flatcorr |
| | 208 | ++ flatcorr.run 0 0 0 0 (dynamic) |
| | 209 | ++ chip.imfile.load 1093 1093 0 0 chiptool |
| | 210 | ++ chip.imfile.run 16 16 0 0 chip_imfile.pl |
| | 211 | ++ chip.advanceexp 166 166 0 0 chiptool |
| | 212 | -+ chip.cleanup.load 0 0 0 0 (dynamic) |
| | 213 | -+ chip.cleanup.run 0 0 0 0 (dynamic) |
| | 214 | ++ camera.exp.load 1090 1089 0 0 camtool |
| | 215 | ++ camera.exp.run 16 16 0 0 camera_exp.pl |
| | 216 | -+ camera.cleanup.load 0 0 0 0 (dynamic) |
| | 217 | -+ camera.cleanup.run 0 0 0 0 (dynamic) |
| | 218 | ++ fake.imfile.load 1102 1102 0 0 faketool |
| | 219 | ++ fake.imfile.run 16 16 0 0 fake_imfile.pl |
| | 220 | ++ fake.advanceexp 165 165 0 0 faketool |
| | 221 | -+ fake.cleanup.load 0 0 0 0 (dynamic) |
| | 222 | -+ fake.cleanup.run 0 0 0 0 (dynamic) |
| | 223 | ++ warp.exp.load 1082 1082 0 0 warptool |
| | 224 | ++ warp.exp.run 16 16 0 0 warp_overlap.pl |
| | 225 | ++ warp.skycell.load 1098 1098 0 0 warptool |
| | 226 | ++ warp.skycell.run 32 32 0 0 warp_skycell.pl |
| | 227 | ++ warp.advancerun 163 163 0 0 warptool |
| | 228 | -+ warp.cleanup.load 0 0 0 0 (dynamic) |
| | 229 | -+ warp.cleanup.run 0 0 0 0 (dynamic) |
| | 230 | ++ diff.skycell.load 1085 1085 0 0 difftool |
| | 231 | ++ diff.skycell.run 16 16 0 0 diff_skycell.pl |
| | 232 | -+ diff.cleanup.load 0 0 0 0 (dynamic) |
| | 233 | -+ diff.cleanup.run 0 0 0 0 (dynamic) |
| | 234 | ++ stack.skycell.load 1097 1097 0 0 stacktool |
| | 235 | ++ stack.skycell.run 3 3 0 0 stack_skycell.pl |
| | 236 | -+ stack.cleanup.load 0 0 0 0 (dynamic) |
| | 237 | -+ stack.cleanup.run 0 0 0 0 (dynamic) |
| | 238 | |
| | 239 | Jobs in Pantasks Queue |
| | 240 | 5: detrend.reject.load 4822 dettool (68ec60) |
| | 241 | 5: detrend.normexp.load 4823 dettool (692030) |
| | 242 | 5: camera.exp.load 4824 camtool (693870) |
| | 243 | 5: detrend.processexp.load 4825 dettool (68f410) |
| | 244 | 5: detrend.process.load 4826 dettool (692560) |
| | 245 | }}} |
| | 246 | |
| | 247 | The number of jobs in stack.skycell.run varies from 2 to 4. |
| | 248 | |
| | 249 | The pantask server will keep running and loading jobs. To terminate the server use "halt"; to pause the server use "stop"; to restart it and resume job execution use "run"; and to kill the server use "quit". |
| | 250 | |
| | 251 | == Using ippMonitor to look at simtest results == |
| | 252 | |
| | 253 | Your simtest results can be viewed using the ippMonitor. Outside the Pan-STARRS network the ippMonitor for the Manoa cluster can be accessed at: http://alala.ifa.hawaii.edu/ippMonitor/ From inside it is: http://alala.ifa.hawaii.edu/ippMonitor/ The MHPCC cluster is only available from inside the network at: http://ipp004.ifa.hawaii.edu/ippMonitor/ A user name and password is needed for each site (independent of the other). |
| | 254 | |
| | 255 | === To see your database it has to be added to the ippMonitor using the command: === |
| | 256 | {{{ |
| | 257 | ippadmin project (dbserver) (dbuser) (dbname) |
| | 258 | create a new ipp project (database) |
| | 259 | }}} |
