​movie rad ​shockwave video games ​inspirational peoms ​roxana diaz y jorge reyes video ​microsoft proofing tools torrent ​pocket video ​nba 2k6 video ​inspirational quote sister ​anal full length video ​sitemap drontar Notes on running IPP on SDSS stripe 82 multi-epoch data

Sebastian Jester jester at mpia.de

Use these as a guide for running IPP for the first time on data from a new camera.

Questions

# At what stage is the fake object insertion/dection run? # Is warptool, stacktool, difftool operation going to be automated?

To-dos

# Look up the distortion correction for SDSS images and whether its WCS is good enough or not (ask Hubert Lampeitl?)

Useful for everything

To get more error messages to a command-line invocation of an IPP executable, add

-trace err 10

Step 0: Get a working installation of IPP

Gene set me up at IPP; there's a working version on panapple at MPIA, and I'm working on getting one set up on aida77-80.

See installing and using multiple IPP versions in parallel

Step 1: set up config files

See Configfiles.

Step 2: Run psphot on single SDSS image

First I create a file firstimg.lis which has the name of the input file. Then I run psphot thus:

psphot -list firstimg.lis firsttry

I could also have given it the input image on the command line with

psphot -file input.fits firsttry

firsttry is the rootname for the output. Here's what happens when I run psphot on ​fpC-001056-i1-0192.fit

Camera SDSS, format SDSS matches header. Chip 0: 1 1 Cell 0: 1 1 Cell 0: 1 1 PHOTCODE is SDSS.i.21 System random seed value used seed = 4479799321607236930 hex image sky : mean 1308.983600 stdev 10.202114 background sky : min 890.523438 mean 1308.983154 max 1349.394043 stdev 32.456470 subtracted background model: 0.283947 sec built smoothed signficance image: 0.824891 sec 104 peaks: 0.104587 sec 100 sources, 100 moments, 4 failed: 0.116634 sec found 0 satstar blend peaks, leaving 100 sources: 0.000100 sec identified 11 blended objects: 0.000904 sec psf clump X, Y: 1.646315, 1.765633 psf clump DX, DY: 0.056862, 0.044356 SN range (moments): 26.758667 - 1668.058228 SN range (peaks) : 26.420599 - 1264.513428 rough classification: 0.000428 sec selected candidate 64 PSF objects fit ext: 1.151582 sec for 63 of 64 sources fit psf: 0.476687 sec for 45 of 64 sources chisq vs flux fit: 0.006806 +/- 0.227294 chisq vs flux fit term 0: 0.035698 +/- 0.200890 chisq vs flux fit term 1: 2.193609 +/- 0.292923 apresid: -0.018228 +/- 0.021941; keeping 29 of 64 psf stars try model PS_MODEL_QGAUSS, ap-fit: -0.018228 +/- 0.021941 : sky bias: 0.000000 dSxo: mean: 0.003162, median: 0.001405, stdev: 0.013455, npts: 29 dSyo: mean: 0.000718, median: 0.002953, stdev: 0.012298, npts: 29 ndSx: mean: 0.045632, median: -0.045638, stdev: 1.276208, npts: 29 ndSy: mean: -0.089024, median: 0.223447, stdev: 1.087374, npts: 29 generate residuals for 29 objects: 0.270703 sec select psf model: 1.917943 sec psf model PS_MODEL_QGAUSS, ApResid: -0.018228 +/- 0.021941 no external sources supplied built models for 100 objects: 0.232271 sec measure ensemble of PSFs: 0.396416 sec GrowthCurve : apLoss : 0.012242 measure aperture residuals for 64 objects (2 skipped, 0 failed, 34 invalid) measure full-frame aperture residuals for 27 of 64 objects: 0.879843 sec aperture residual: -0.012413 +/- 0.019795 : 0.000000 bias, 0.000000 skysat measure magnitudes : 0.558418 sec for 100 objects (86 with apertures) replace background flux : 0.056657 sec creating moments plot saving plot to firsttry.mnt.png creating psf model plot saving plot to firsttry.psf.png creating ap mag - psf mag plot saving plot to firsttry.dap.png complete psphot run: 8.708983 sec

found 0 leaks at psphot

I also tried this on the Mac in Heidelberg and get different results...:

Camera SDSS, format SDSS matches header. Chip 0: 1 1 Cell 0: 1 1 Cell 0: 1 1 PHOTCODE is SDSS.i.21 System random seed value used seed = 2726282019464764904 hex image sky : mean 1311.806081 stdev 8.951074 background sky : min 1305.236572 mean 1311.805908 max 1372.219238 stdev 8.360644 subtracted background model: 0.268568 sec built smoothed signficance image: 0.662050 sec 99 peaks: 0.081675 sec 96 sources, 96 moments, 3 failed: 0.094184 sec found 0 satstar blend peaks, leaving 96 sources: 0.000078 sec identified 8 blended objects: 0.007648 sec psf clump X, Y: 1.645948, 1.765079 psf clump DX, DY: 0.056189, 0.045065 SN range (moments): 26.627832 - 1668.060669 SN range (peaks) : 26.431179 - 1264.521973 rough classification: 0.012332 sec selected candidate 64 PSF objects fit ext: 0.840499 sec for 63 of 64 sources fit psf: 0.416352 sec for 42 of 64 sources chisq vs flux fit: 0.005232 +/- 0.371809 chisq vs flux fit term 0: -0.088947 +/- 0.187928 chisq vs flux fit term 1: 2.666661 +/- 0.175476 apresid: -0.037835 +/- 0.032469; keeping 26 of 64 psf stars try model PS_MODEL_QGAUSS, ap-fit: -0.037835 +/- 0.032469 : sky bias: 0.000000 dSxo: mean: 0.001541, median: -0.001044, stdev: 0.014191, npts: 26 dSyo: mean: 0.001470, median: 0.001485, stdev: 0.011560, npts: 26 ndSx: mean: 0.125409, median: -0.191685, stdev: 1.329126, npts: 26 ndSy: mean: -0.105611, median: 0.018051, stdev: 1.013325, npts: 26 generate residuals for 26 objects: 0.283203 sec select psf model: 1.554244 sec psf model PS_MODEL_QGAUSS, ApResid: -0.037835 +/- 0.032469 no external sources supplied built models for 96 objects: 0.170146 sec measure ensemble of PSFs: 0.447365 sec GrowthCurve : apLoss : 0.008937 measure aperture residuals for 64 objects (3 skipped, 0 failed, 29 invalid) measure full-frame aperture residuals for 28 of 64 objects: 0.845621 sec aperture residual: -0.012332 +/- 0.020148 : 0.000000 bias, 0.000000 skysat measure magnitudes : 0.417724 sec for 96 objects (85 with apertures) replace background flux : 0.027076 sec skipping moments plot skipping psf model plot skipping ap-mag resid plot complete psphot run: 5.548932 sec

I thought I was using exactly the same settings (copied all the config files 1:1 from IfA to MPIA), but maybe not? E.g. why is it skipping the moments plot in HD, but not in HI? According to Gene, the different numbers are explicable by different random numbers (i.e. seeds). The plots missing would only make sense if kapa wasn't compiled properly, but we think it was.

Step 3: inserting results into dvo and inspecting them

psphot produces its output catalog in firsttry.cmf which we load into dvo:

addstar -D CAMERA sdss -accept-astrom firsttry.cmf

-accept-astrom says that dvo should trust the WCS and ignore the fact that there are 0 astrometric stars reported in the header. Ohana commands like addstar require the -D CAMERA flag on the command line.

Then start dvo by typing 'dvo' and plot things:

# Make a plot window region 0 0 90 ait cgrid current style -x 2 -c black -pt 2 -lt 0 -lw 0.500000 -sz 1.000000 # The above line is the output from csgrid. It's not something to type in. # (May need to repeat both commands after adjusting size of kapa plot window)

# tell it where the catalogs are catdir /data/ipp001.0/jester/stripe82/coadd/catdir

# make a new display for displaying images images -n 1

# read a fits image into a buffer buf0 rd buf0 /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0192.fit

# display it with cuts given tv buf0 1300 3000 # in the TV window, click left to pan, middle to zoom out, right to zoom in

# Tell IPP which region of the sky I'm interested in skyregion 350 355 -2 0 # Extract individual measurements that exist in catdir for that region mextract ra,dec,mag # Show how many entries each vector has vectors

N name size 0 ra 100 1 dec 100 2 mag 100

# convert ra,dec positions into display's x,y positions in place coords buf0 -c ra dec # and overlay the object positions on the display window vload red ra dec # To just plot the ra,dec on a coordinate grid (i.e., not an image), say cplot ra dec

Step 4: warping and stacking

Set up a grid of skycells to which the output will be warped

DVO has Gene's tesselation of the sky built in. To create a table with the skycells of this tesselation, there's the skycells executable:

skycells 8 -scale .3 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/

The '8' tells it the tiling level, where 1 is coarse and 10 is fine. -scale gives the arcseconds per pixel. The skycells table is put into the directory specified by CATDIR.

To look at the resulting tesselation, bring up dvo:

# Set catdir to the directory where the skycells live catdir /data/ipp001.0/jester/stripe82/coadd/skycells/ # Plot them up, first set up the grid (args are ra dec radius projection) region 352 -1 2 ait images # Overplot the outline of a fits image imbox -c red /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0193.fit # Same for a list of images - first read list into a list variable list myImlist -x "ls /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fit" # Plot them up in a for loop for i 0 $myImlist:n

imbox -c red $myImlist:$i end

# The preceding was for individual fits files; if dvo already knows about images (via addstar), it works like this: # set catdir to place where images are catdir /data/ipp001.0/jester/stripe82/coadd/catdir/ # plot them up images -c red

# To set up the next step, we need to know the id of the skycell. This can be done semi-interactively: cursor # then move the cursor into the skycell whose number you want to know and press any letter key, # e.g. d, to store the ra,dec of the point where you press the key (q to quit, also stores things in q). # $Rq and $Dq then store the ra,dec of that point, and you can use them in 'gimages' to get the skycell id: gimages $Rq $Dq -region

output-> reading images (mode DVO_MODE_MEF) output-> 0 skycell.0484667 353.0672 -1.1307 1970/01/01,00:00:00 0 0 0.00 0.000 0.000 0.000 1

skycell.0484667 is the ID we need.

Warp images to the skycell grid

Setup: create a pixelless fits image containg the WCS of the skycell

This uses a shell command:

dvoImageExtract skycell.0484667 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ -o /data/ipp001.0/jester/stripe82/coadd/run_ipp/skycell_0484667_warptemplate.fits

to create the "warp template" fits file. It's a fits image with NAXIS=0, i.e., no pixels - just the header.

Run psWarp

You need to have the config files for the "simple" camera to run psWarp as follows:

pswarp -file ../input/fpC-001056-i1-0192.fit psWarp-001056-i1-0192 skycell_0484667_warptemplate.fits pswarp -file ../input/fpC-001755-i1-0330.fit psWarp-001755-i1-0330 skycell_0484667_warptemplate.fits

The 'output' parameter gives a rootname, so no .fits needed - it gets added automatically. The skycell needs to be specified with its full pathname.

Stack the images with ppStack

To run ppStack, you need an input file that specifies input images, weights, masks, seeing; here's an (the .txt is just there because the wiki judges files by its name and doesn't know .mdc files). You then simply say:

ppStack ppStack.mdc ppStack-run1

where ppStack-run1 is the output rootname.

Step 5: run psphot on the stacked image

Initially, this barfed with some config file issue, but Paul fixed it. Now it runs with

psphot -file ppStack-run1.fits photstack-run1

Step 6: inject stacked psphot results into dvo

Step 7 (really chapter 2): run everything in automated fashion in pantasks

Preliminaries: set up a database in ippMon

The db administrator uses ippMonitor/dbadmin to create a new "project" in the mySQL database (this isolates different reduction runs from each other; I get the impression that it's useful to really have one project per run, as the easiest [and perhaps only?] way to start over with something is to delete all the tables in the project and start from scratch).

Administrator things

If the database is being set up for the very first time at all for IPP use, the admin need sto say (e.g.)

dbadmin init (dbserver) (dbname) (dbpassword) dbadmin init aida77 ipp

This creates an administrative database ippadmin which keeps track of everything else and is just required once per IPP installation.

To set up a new project database, then say

dbadmin project (dbserver) (dbuser) (dbname) dbadmin project aida77 ipp jester_s82

confirming with the password set above. This creates a project database jester_s82 that keeps track of all files in that project.

There's also something about creating a www user:

dbadmin user (dbserver) (dbuser) (username) (password)

but I'm still fighting with the web server. NB: in our case, the php script install location is

/srv/www/htdocs/ippMonitor

User things

The server, ipp-dbname, password and project are specified in ~/.ipprc and/or site.config. I have in site.config:

### Database configuration DBSERVER STR aida77 # Database host name (for psDBInit) DBUSER STR ipp # Database user name (for psDBInit) DBPASSWORD STR * # Database password (for psDBInit)

and in ~/.ipprc:

DBNAME STR jester_s82 # Database name (for psDBInit)

I as user then use pxadmin to create a new database, or delete or recreate it (make sure that DBNAME is pointing to the database you actually want to work on!)

pxadmin -create

There should proabably be something like an "IPP administrator's guide" (or maybe there is already one) that spells out how mySQL should be set up, what the best way is to make sure that different users don't interfere with each other's tables and projects (e.g. by having multiple mySQL users, if possible), etc.

To start a whole reduction afresh, at the moment the fastest/only way is

pxadmin -recreate

This will delete EVERYTHING in your database DBNAME...

Injecting and registering frames in ippMon

Now we need to tell ippMon about our images. This has been covered in ] and also in ​Gene and Paul et al's giant 2-day presentation on IPP from the Baltimore workshop (slide 228).

Injection means telling IPP about the files

Adding a static bad pixel mask to the db

I've written a path://S82/sdssBPmask/registerBPmasks.pl that runs the injection of the bad-pixel masks, first defining a detrend run for each filter, then injecting all the I created from the . The bad pixel masks only work with the mosaiced camera below - in fact, they were the motivation for introducing the mosaiced SDSS camera.

After pxadmin recreate, the bad-pixel masks need to be injected before the image files, so they are there and can be used'''

Injection with the "single chip = FPA" config

This is in

~jester/IPP/ippconfig/sdss/camera.config

The injection is simple here:

ipp_serial_inject.pl --telescope sdss --camera sdss \ --workdir path://S82/run_ipp --dbname jester \ --reduction PROCESSED /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fits

The "reduction class" PROCESSED selects a given set of reduction steps that is defined in the ppImage.config file. In the sdssmosaic/ppImage.config, I renamed this class into SDSSphotonly, which runs psphot but not psastro, to distinguish it from SDSSphotast, which runs both.

Injection with the "six chips = mosaiced FPA" config

This is in

~jester/IPP/ippconfig/sdssmosaic/

and the camera name is SDSSmosaic. Whether an SDSS image is recognized as SDSSmosaic or the single-chip SDSS format depends on which of the lines in my ~/.ipprc is commented out - the format.config for both files look identical, since the files are identical.

The injection is more involved, since the injection needs to know about the grouping. There are two possibilities: ipp_serial_inject_mosaic.pl and ipp_serial_inject_split.pl

The former needs to be modified to know about the structure of SDSS. It then needs to be called once per group of 6 files that are part of the same "exposure".

The latter needs to have the input files put into subdirectories, with one dir per "exposure". You call it with a glob for all the directories [I think].

Since I need to write a script either way, either to copy to directories or to call with a suitable glob per exposure, I'll take the latter route so we don't need to change ipp_serial_inject_mosaic.pl.

To inject all the files in a given directory $D/stripe82/coadd/input/i/1056-0192/, use this command:

ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ --workdir path://S82/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82CATDIR \ $D/data/SDSS/stripe82/coadd/input/i/1056-0192/

tess_id and dvodb are new in 2.5. The tesselation_name and dvodb_name can be explicit paths to catdirs above, but also abstract names defined in site.config

SDSSphotonly corresponds to the PROCESSED reduction class (now PPIMAGE_OP_SOFT) defined in sdss[mosaic]/camara.config using the recipes in sdss[mosaic]/ppImage.config. The other new reduction class is SDSSphotast, which includes astrometry.

For the initial testing, I want to isolate any possible astrometry problems from photometry problems, so I'll continue to work with SDSSphotonly. However, I wanted to make sure that phot & ast both run through, so that I can rectify any mechanical problems while still at the IfA, and then worry only about functional problems later.

Registering

"Registering" means getting them ingested into ippMon:

# Start pantasks pantasks # Inside pantasks, load predefined procuedures module pantasks.pro module.tasks # Add host names for parallel processing controller host add ipp000 # etc. for more hosts. # Now by default, all analysis steps are on by default, but we want # to go through them step by step to make sure everything's working. # Hence we turn 'all' off, and then turn things on selectively. # Also, at this stage, IPP doesn't know by itself that fpC*.fits images # from SDSS don't need detrending, so we need to tell IPP by hand after the # register step. all.off register.on run

You can also save these steps in a macro, by puttin the commands in a file, wrapping them in "macro myMacro" and "end", and then reading the file in pantasks with the "input" command. In pantasks, one can then call the macro by just typing myMacro. On the IfA cluster, my macro that does these things is called thus:

input /home/panstarrs/jester/IPP/macros/inject.pro injectmine

Note that it is possible to 'controller add' the same host multiple times, and it will get multiple shares of the load.

Processing things with the pipeline

All the reduction steps are just switches in the pantasks.pro module; they can be run simply by turning them on, e.g.

chip.on camera.on

runs psphot.

The next step is warping, which requires some extra command-line work (but maybe there's a way to have IPP do this, too? There has to be one, eventually...)

Warping

Defining warp runs

In IPP 2.5, there's a new feature where you specify the skycell tessellation id already in the injection step (see above). You then just need to turn on the warping step, and it figures out what to warp where by itself, and does it.

Run them

Then

warp.on run

will run the actual warping. To see what's going on, say

status

for a general job count, and

status -taskstats

to see statistics on execution times.

Stacking

Now that everything is warped, we could stack it. For this, we need to define a 'stack run' with stacktool.

stacktool -definerun -workdir path://S82/run_ipp -skycell_id skycell.0472823 -warp_id 1 -warp_id 2 -warp_id 5 -tess_id Skycells

This tells IPP to stack things in the skycell given and include the pictures from the warp runs listed. The tess_id is defined in the TESSELLATIONS METADATA in camera.config, and points to the same path that I used to set up the tessellation above

At the moment, you need to figure out yourself which skycells and warp_ids to use. I do this:

# First, connect to the database mysql -h alala -u ipp -p jester [enter password] # then run the following query

select distinct skycell_id, warp_id from warpSkyCellMap where skycell_id in (select skycell_id from warpSkyCellMap

group by skycell_id having count(distinct warp_id) > 1)

order by skycell_id, warp_id;

The inner query shows you how many warp runs overlap a given skycell:

select skycell_id, count(distinct warp_id) as N_warp, count(*) as N_img from warpSkyCellMap group by skycell_id having N_img > 1 order by skycell_id ;

Conversely, you can find out how many skycells each of your images was split into:

select cam_id, class_id, warp_id, count(distinct skycell_id) from warpSkyCellMap group by class_id, warp_id, cam_id order by cam_id, class_id, warp_id;

If something fails, the way to get IPP to stop trying is

stacktool -updaterun -stack_id 2 -state stop

Diffing

In principle, we could also define a diff run with difftool, but since I'm not interested in image subtraction myself, at least not initially (sorting out brown dwarfs via proper motion may change this in the future), I am skipping this step.

However, the SDSS supernova survey has been preparing high-quality subtractions for this same data set, so it would be a very good comparison set, too. Gajus Miknaitis at Fermilab knows about the SDSS set, and I'm happy to help establishing contacts if necessary.

Running pshot on the stack

Don't know how that works in the pipeline yet.

Loading stacked psphot outputs into dvo

Should work as above, in principle.

Log 2008-05-28

I don't know where I'm at on aida77 so I'm starting afresh.

Creating a database

Start with fresh jester_s82 reduction database:

pxadmin -recreate -dbname jester_s82

or create a new one for a new run:

ippadmin project aida77 ipp jester_s82_20080912 pxadmin -create -dbname jester_s82_20080912

Injecting

Normally I'd want to inject the bad pixel masks as described above, but I'm skipping that for now. Instead, go straight to injection, for now just with all the fields from 2 runs to see that it works:

ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ --workdir path://S82TEST/run_ipp --dbname jester_s82_20080912 --tess_id S82CELLS --dvodb S82TESTCATDIR \ /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/

or for everything:

ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ --workdir path://S82PROD/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82PRODCATDIR \ /IPP/data/SDSS/stripe82/coadd/input/?/*/

The symbolic names like S82TEST are defined in <code>~/.ipprc</code> or <code>/IPP/site.config</code> (or another site config file pointed to by your <code>~/.ipprc</code>).

With different recipes

At some point, IPP allowed using recipes also for psphot, which are in sdssmosaic/psphot.config and sdssmosaic/reductionClasses.mdc For these, use things like

pxadmin -recreate -dbname jester_s82_20081217_ap3

ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic \ --reduction SDSSphotast_ap \ --workdir path://S82AP3_20081217 \ --dbname jester_s82_20081217_ap3 \ --tess_id S82CELLS \ --dvodb S82AP3CAT_20081217 \ /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/

This recipe turns on both aperture corrections and spatially variable PSF fits.

Pantasksing

Then start jobs in pantasks:

pantasks # If it's not in your ~/.pantasksrc module pantasks.pro module.tasks

# There's also a file with the sort of commands that I'm describing here # in /home/jester/s82.pro which you can load with input /home/jester/s82.pro

# Set database if different from DBNAME in ~/.ipprc add.database jester_s82_20080912

# Turn all reduction steps off, except for register, chip and camera: all.off register.on chip.on camera.on

# Add some CPUs for the task scheduler to use # - need to be able to ssh without passwords to the host given # (see /IPP/README) controller host add aida77 -threads 2

# This command can be issued multiple times for adding more # CPUs on the same machine, or more machines. # The -threads 2 is optional and allows multi-threading, # which should be the fastest option. # aida77 (and aida78 etc.) have 2 dual-core CPUs. Hence: # Without the -threads, use at most 3 copies of aida77 # (fewer if more than one person are running pantasks!) # With the threads, use at most 2 copies of aida77.

# Now start the processing run

# say 'stop' to stop launching new jobs, or 'halt' to # stop processing altogether (and being flooded by error messages)

# To see what's going on: controller status # lists active and pending jobs and the status of the hosts

status # lists pending jobs and statistics of success/failures

status -taskstats # includes information on how long each job ran

NB: with

$VERBOSE = 2

in pantasks, you see the commands as they're issued, which is useful for diagnosing problems. You can also simply re-run the command that's announced as failing, or look at the log file indicated. Normally, to re-run without generating further error messages, you need to remove the <code>--log path://datata</code> argument to see the output (which otherwise goes into a log file), and add <code>--no-update</code> so that the database isn't affected by the manual repeat.

Once the camera-stage processing is finished, there's a dvo catalog with the outputs in S82PRODCATDIR. This needs to be sorted:

cd ipp_datapath.pl path://S82PRODCATDIR addstar -D CATDIR . -resort relphot -D CATDIR . -averages

The resorted version is available for download at ​http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz - a tarfile of about 250 MB that expands into a directory of about 500 MB. I have asked our computer people also to make available all the IPP outputs directly; however, those are 250 GB (no compression) so they need to figure out a way to host those. The input fpC*.fits images from SDSS are another ~50 GB. Here is the .tgz