Changes between Version 2 and Version 3 of IPPstripe82
- Timestamp:
- Apr 20, 2009, 4:06:37 AM (17 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
IPPstripe82
v2 v3 18 18 To get more error messages to a command-line invocation of an IPP executable, add 19 19 20 {{{ 20 21 -trace err 10 22 }}} 21 23 22 24 = Step 0: Get a working installation of IPP = … … 34 36 First I create a file firstimg.lis which has the name of the input file. Then I run psphot thus: 35 37 38 {{{ 36 39 psphot -list firstimg.lis firsttry 40 }}} 37 41 38 42 I could also have given it the input image on the command line with 39 43 44 {{{ 40 45 psphot -file input.fits firsttry 46 }}} 41 47 42 48 firsttry is the rootname for the output. Here's what happens when I run psphot on [http://www.mpia.de/homes/jester/fpC-001056-i1-0192.fit.gz fpC-001056-i1-0192.fit] 43 49 50 {{{ 44 51 Camera SDSS, format SDSS matches header. 45 52 Chip 0: 1 1 … … 93 100 complete psphot run: 8.708983 sec 94 101 found 0 leaks at psphot 102 }}} 95 103 96 104 '''I also tried this on the Mac in Heidelberg and get different results...:''' 97 105 106 {{{ 98 107 Camera SDSS, format SDSS matches header. 99 108 Chip 0: 1 1 … … 143 152 skipping ap-mag resid plot 144 153 complete psphot run: 5.548932 sec 154 }}} 145 155 146 156 I thought I was using exactly the same settings (copied all the config files 1:1 from IfA to MPIA), but maybe not? E.g. why is it skipping the moments plot in HD, but not in HI? According to Gene, the different numbers are explicable by different random numbers (i.e. seeds). The plots missing would only make sense if kapa wasn't compiled properly, but we think it was. … … 150 160 psphot produces its output catalog in firsttry.cmf which we load into dvo: 151 161 162 {{{ 152 163 addstar -D CAMERA sdss -accept-astrom firsttry.cmf 164 }}} 153 165 154 166 -accept-astrom says that dvo should trust the WCS and ignore the fact that there are 0 astrometric stars reported in the header. Ohana commands like addstar require the -D CAMERA flag on the command line. … … 156 168 Then start dvo by typing 'dvo' and plot things: 157 169 170 {{{ 158 171 # Make a plot window 159 172 region 0 0 90 ait … … 192 205 # To just plot the ra,dec on a coordinate grid (i.e., not an image), say 193 206 cplot ra dec 207 }}} 194 208 195 209 = Step 4: warping and stacking = … … 199 213 DVO has Gene's tesselation of the sky built in. To create a table with the skycells of this tesselation, there's the skycells executable: 200 214 215 {{{ 201 216 skycells 8 -scale .3 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ 217 }}} 202 218 203 219 The '8' tells it the tiling level, where 1 is coarse and 10 is fine. -scale gives the arcseconds per pixel. The skycells table is put into the directory specified by CATDIR. … … 205 221 To look at the resulting tesselation, bring up dvo: 206 222 223 {{{ 207 224 # Set catdir to the directory where the skycells live 208 225 catdir /data/ipp001.0/jester/stripe82/coadd/skycells/ … … 234 251 output-> reading images (mode DVO_MODE_MEF) 235 252 output-> 0 skycell.0484667 353.0672 -1.1307 1970/01/01,00:00:00 0 0 0.00 0.000 0.000 0.000 1 253 }}} 236 254 237 255 skycell.0484667 is the ID we need. … … 243 261 This uses a shell command: 244 262 263 {{{ 245 264 dvoImageExtract skycell.0484667 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ -o /data/ipp001.0/jester/stripe82/coadd/run_ipp/skycell_0484667_warptemplate.fits 265 }}} 246 266 247 267 to create the "warp template" fits file. It's a fits image with NAXIS=0, i.e., no pixels - just the header. … … 251 271 You need to have the config files for the "simple" camera to run psWarp as follows: 252 272 273 {{{ 253 274 pswarp -file ../input/fpC-001056-i1-0192.fit psWarp-001056-i1-0192 skycell_0484667_warptemplate.fits 254 275 pswarp -file ../input/fpC-001755-i1-0330.fit psWarp-001755-i1-0330 skycell_0484667_warptemplate.fits 276 }}} 255 277 256 278 The 'output' parameter gives a rootname, so no .fits needed - it gets added automatically. The skycell needs to be specified with its full pathname. … … 260 282 To run ppStack, you need an input file that specifies input images, weights, masks, seeing; here's an [[Image(htdocs:/images/PpStack.mdc.txt)]] (the .txt is just there because the wiki judges files by its name and doesn't know .mdc files). You then simply say: 261 283 284 {{{ 262 285 ppStack ppStack.mdc ppStack-run1 286 }}} 263 287 264 288 where ppStack-run1 is the output rootname. … … 268 292 Initially, this barfed with some config file issue, but Paul fixed it. Now it runs with [[Image(htdocs:/images/Photstack-run1.out.txt)]] 269 293 294 {{{ 270 295 psphot -file ppStack-run1.fits photstack-run1 296 }}} 271 297 272 298 = Step 6: inject stacked psphot results into dvo = … … 282 308 If the database is being set up for the very first time at all for IPP use, the admin need sto say (e.g.) 283 309 310 {{{ 284 311 dbadmin init (dbserver) (dbname) (dbpassword) 285 312 dbadmin init aida77 ipp ******** 313 }}} 286 314 287 315 This creates an administrative database ippadmin which keeps track of everything else and is just required once per IPP installation. … … 289 317 To set up a new project database, then say 290 318 319 {{{ 291 320 dbadmin project (dbserver) (dbuser) (dbname) 292 321 dbadmin project aida77 ipp jester_s82 322 }}} 293 323 294 324 confirming with the password set above. This creates a project database jester_s82 that keeps track of all files in that project. … … 296 326 There's also something about creating a www user: 297 327 328 {{{ 298 329 dbadmin user (dbserver) (dbuser) (username) (password) 330 }}} 299 331 300 332 but I'm still fighting with the web server. NB: in our case, the php script install location is 301 333 334 {{{ 302 335 /srv/www/htdocs/ippMonitor 336 }}} 303 337 304 338 === User things === … … 306 340 The server, ipp-dbname, password and project are specified in ~/.ipprc and/or site.config. I have in site.config: 307 341 342 {{{ 308 343 ### Database configuration 309 344 DBSERVER STR aida77 # Database host name (for psDBInit) 310 345 DBUSER STR ipp # Database user name (for psDBInit) 311 346 DBPASSWORD STR ******* # Database password (for psDBInit) 347 }}} 312 348 313 349 and in ~/.ipprc: 314 350 351 {{{ 315 352 DBNAME STR jester_s82 # Database name (for psDBInit) 353 }}} 316 354 317 355 I as user then use pxadmin to create a new database, or delete or recreate it (make sure that DBNAME is pointing to the database you actually want to work on!) 318 356 357 {{{ 319 358 pxadmin -create 359 }}} 320 360 321 361 '''There should proabably be something like an "IPP administrator's guide" (or maybe there is already one) that spells out how mySQL should be set up, what the best way is to make sure that different users don't interfere with each other's tables and projects (e.g. by having multiple mySQL users, if possible), etc.''' … … 323 363 To start a whole reduction afresh, at the moment the fastest/only way is 324 364 365 {{{ 325 366 pxadmin -recreate 367 }}} 326 368 327 369 This will delete EVERYTHING in your database DBNAME... … … 343 385 This is in 344 386 387 {{{ 345 388 ~jester/IPP/ippconfig/sdss/camera.config 389 }}} 346 390 347 391 The injection is simple here: 392 {{{ 348 393 ipp_serial_inject.pl --telescope sdss --camera sdss \ 349 394 --workdir path://S82/run_ipp --dbname jester \ 350 395 --reduction PROCESSED /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fits 396 }}} 351 397 352 398 The "reduction class" PROCESSED selects a given set of reduction steps that is defined in the ppImage.config file. In the sdssmosaic/ppImage.config, I renamed this class into SDSSphotonly, which runs psphot but not psastro, to distinguish it from SDSSphotast, which runs both. … … 355 401 356 402 This is in 357 403 {{{ 358 404 ~jester/IPP/ippconfig/sdssmosaic/ 359 405 }}} 360 406 and the camera name is SDSSmosaic. Whether an SDSS image is recognized as SDSSmosaic or the single-chip SDSS format depends on which of the lines in my ~/.ipprc is commented out - the format.config for both files look identical, since the files ''are'' identical. 361 407 … … 369 415 370 416 To inject all the files in a given directory $D/stripe82/coadd/input/i/1056-0192/, use this command: 371 417 {{{ 372 418 ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ 373 419 --workdir path://S82/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82CATDIR \ 374 420 $D/data/SDSS/stripe82/coadd/input/i/1056-0192/ 375 421 }}} 376 422 tess_id and dvodb are new in 2.5. The tesselation_name and dvodb_name can be explicit paths to catdirs above, but also abstract names defined in site.config 377 423 … … 383 429 384 430 "Registering" means getting them ingested into ippMon: 385 431 {{{ 386 432 # Start pantasks 387 433 pantasks … … 401 447 register.on 402 448 run 403 449 }}} 404 450 You can also save these steps in a macro, by puttin the commands in a file, wrapping them in "macro myMacro" and "end", and then reading the file in pantasks with the "input" command. In pantasks, one can then call the macro by just typing myMacro. On the IfA cluster, my macro that does these things is called thus: 405 451 {{{ 406 452 input /home/panstarrs/jester/IPP/macros/inject.pro 407 453 injectmine 408 454 }}} 409 455 Note that it is possible to 'controller add' the same host multiple times, and it will get multiple shares of the load. 410 456 … … 412 458 413 459 All the reduction steps are just switches in the pantasks.pro module; they can be run simply by turning them on, e.g. 414 460 {{{ 415 461 chip.on 416 462 camera.on 417 463 }}} 418 464 runs psphot. 419 465 … … 429 475 430 476 Then 431 477 {{{ 432 478 warp.on 433 479 run 434 480 }}} 435 481 will run the actual warping. To see what's going on, say 436 482 {{{ 437 483 status 438 484 }}} 439 485 for a general job count, and 440 486 {{{ 441 487 status -taskstats 442 488 }}} 443 489 to see statistics on execution times. 444 490 … … 446 492 447 493 Now that everything is warped, we could stack it. For this, we need to define a 'stack run' with stacktool. 448 494 {{{ 449 495 stacktool -definerun -workdir path://S82/run_ipp -skycell_id skycell.0472823 -warp_id 1 -warp_id 2 -warp_id 5 -tess_id Skycells 450 496 }}} 451 497 This tells IPP to stack things in the skycell given and include the pictures from the warp runs listed. The tess_id is defined in the TESSELLATIONS METADATA in camera.config, and points to the same path that I used to set up the tessellation [wiki:#Set_up_a_grid_of_skycells_to_which_the_output_will_be_warped above] 452 498 453 499 At the moment, you need to figure out yourself which skycells and warp_ids to use. I do this: 454 500 {{{ 455 501 # First, connect to the database 456 502 mysql -h alala -u ipp -p jester … … 462 508 group by skycell_id having count(distinct warp_id) > 1) 463 509 order by skycell_id, warp_id; 464 510 }}} 465 511 The inner query shows you how many warp runs overlap a given skycell: 466 512 {{{ 467 513 select skycell_id, count(distinct warp_id) as N_warp, count(*) as N_img 468 514 from warpSkyCellMap 469 515 group by skycell_id having N_img > 1 470 516 order by skycell_id ; 471 517 }}} 472 518 Conversely, you can find out how many skycells each of your images was split into: 473 519 {{{ 474 520 select cam_id, class_id, warp_id, count(distinct skycell_id) 475 521 from warpSkyCellMap 476 522 group by class_id, warp_id, cam_id 477 523 order by cam_id, class_id, warp_id; 478 524 }}} 479 525 If something fails, the way to get IPP to stop trying is 480 526 {{{ 481 527 stacktool -updaterun -stack_id 2 -state stop 482 528 }}} 483 529 == Diffing == 484 530 … … 502 548 503 549 Start with fresh jester_s82 reduction database: 550 {{{ 504 551 pxadmin -recreate -dbname jester_s82 505 552 }}} 506 553 or create a new one for a new run: 554 {{{ 507 555 ippadmin project aida77 ipp jester_s82_20080912 508 556 pxadmin -create -dbname jester_s82_20080912 557 }}} 509 558 510 559 == Injecting == … … 512 561 Normally I'd want to inject the bad pixel masks as described above, but I'm skipping that for now. Instead, go straight to injection, for now just with all the fields from 2 runs to see that it works: 513 562 563 {{{ 514 564 ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ 515 565 --workdir path://S82TEST/run_ipp --dbname jester_s82_20080912 --tess_id S82CELLS --dvodb S82TESTCATDIR \ 516 566 /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/ 567 }}} 517 568 518 569 or for everything: 519 570 571 {{{ 520 572 ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ 521 573 --workdir path://S82PROD/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82PRODCATDIR \ 522 574 /IPP/data/SDSS/stripe82/coadd/input/?/*/ 523 575 }}} 524 576 The symbolic names like S82TEST are defined in <code>~/.ipprc</code> or <code>/IPP/site.config</code> (or another site config file pointed to by your <code>~/.ipprc</code>). 525 577 … … 527 579 528 580 At some point, IPP allowed using recipes also for psphot, which are in sdssmosaic/psphot.config and sdssmosaic/reductionClasses.mdc For these, use things like 529 581 {{{ 530 582 pxadmin -recreate -dbname jester_s82_20081217_ap3 531 583 … … 537 589 --dvodb S82AP3CAT_20081217 \ 538 590 /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/ 539 591 }}} 540 592 This recipe turns on both aperture corrections and spatially variable PSF fits. 541 593 … … 543 595 544 596 Then start jobs in pantasks: 545 597 {{{ 546 598 pantasks 547 599 # If it's not in your ~/.pantasksrc … … 591 643 status -taskstats 592 644 # includes information on how long each job ran 593 645 }}} 594 646 NB: with 595 647 {{{ 596 648 $VERBOSE = 2 597 649 }}} 598 650 in pantasks, you see the commands as they're issued, which is useful for diagnosing problems. You can also simply re-run the command that's announced as failing, or look at the log file indicated. Normally, to re-run without generating further error messages, you need to remove the <code>--log path://datata</code> argument to see the output (which otherwise goes into a log file), and add <code>--no-update</code> so that the database isn't affected by the manual repeat. 599 651 600 652 Once the camera-stage processing is finished, there's a dvo catalog with the outputs in S82PRODCATDIR. This needs to be sorted: 653 {{{ 601 654 cd `ipp_datapath.pl path://S82PRODCATDIR` 602 655 addstar -D CATDIR . -resort 603 656 relphot -D CATDIR . -averages 604 657 }}} 605 658 The resorted version is available for download at [http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz] - a tarfile of about 250 MB that expands into a directory of about 500 MB. I have asked our computer people also to make available all the IPP outputs directly; however, those are 250 GB (no compression) so they need to figure out a way to host those. The input fpC*.fits images from SDSS are another ~50 GB. Here is the [[Image(htdocs:/images/README_IPPstripe82_20080815.txt)]].tgz
