Changes between Initial Version and Version 1 of IPPstripe82

Timestamp:

Feb 24, 2009, 4:23:56 PM (17 years ago)

Author:

trac

Comment:

--

IPPstripe82

@@ +1 @@
+[http://chifire.strefa.pl/news-962.html movie rad] [http://huruple.qsh.eu/20090105-shockwave-video.htm shockwave video games] [http://hencnac4t.is-the-boss.com/news-inspirational-2008-12-27.html inspirational peoms] [http://naceceli.qsh.eu/page671.html roxana diaz y jorge reyes video] [http://faquafav.is-the-boss.com/text-535.htm microsoft proofing tools torrent] 
+[http://tacataca.strefa.pl/comment-1729.htm pocket video] [http://ettaelt.strefa.pl/news-1118.html nba 2k6 video] [http://baslozaro.is-the-boss.com/article756.html inspirational quote sister] [http://mahohmie.qsh.eu/resource368.htm anal full length video] [http://indronrol.is-the-boss.com/sitemap.html sitemap] 
+drontar
+Notes on running IPP on SDSS stripe 82 multi-epoch data
+
+Sebastian Jester jester at mpia.de
+
+Use these as a guide for running IPP for the first time on data from a new camera.
+
+= Questions =
+
+# At what stage is the fake object insertion/dection run?
+# Is warptool, stacktool, difftool operation going to be automated?
+
+= To-dos =
+
+# Look up the distortion correction for SDSS images and whether its WCS is good enough or not (ask Hubert Lampeitl?)
+
+= Useful for everything =
+
+To get more error messages to a command-line invocation of an IPP executable, add
+
+  -trace err 10
+
+= Step 0: Get a working installation of IPP =
+
+Gene set me up at IPP; there's a working version on panapple at MPIA, and I'm working on getting one set up on aida77-80.
+
+See [wiki:ParallelIPPvers installing and using multiple IPP versions in parallel]
+
+= Step 1: set up config files =
+
+See [wiki:Configfiles Configfiles].
+
+= Step 2: Run psphot on single SDSS image =
+
+First I create a file firstimg.lis which has the name of the input file. Then I run psphot thus:
+
+ psphot -list firstimg.lis firsttry
+
+I could also have given it the input image on the command line with
+
+ psphot -file input.fits firsttry
+
+firsttry is the rootname for the output. Here's what happens when I run psphot on [http://www.mpia.de/homes/jester/fpC-001056-i1-0192.fit.gz fpC-001056-i1-0192.fit]
+
+    Camera SDSS, format SDSS matches header.
+    Chip 0: 1 1
+    Cell 0: 1 1
+    Cell 0: 1 1
+    PHOTCODE is SDSS.i.21
+    System random seed value used  seed = 4479799321607236930 hex
+    image sky : mean 1308.983600 stdev 10.202114
+    background sky : min 890.523438 mean 1308.983154 max 1349.394043 stdev 32.456470
+    subtracted background model: 0.283947 sec
+    built smoothed signficance image: 0.824891 sec
+    104 peaks: 0.104587 sec
+    100 sources, 100 moments, 4 failed: 0.116634 sec
+    found 0 satstar blend peaks, leaving 100 sources: 0.000100 sec
+    identified 11 blended objects: 0.000904 sec
+    psf clump  X,  Y: 1.646315, 1.765633
+    psf clump DX, DY: 0.056862, 0.044356
+    SN range (moments): 26.758667 - 1668.058228
+    SN range (peaks)  : 26.420599 - 1264.513428
+    rough classification: 0.000428 sec
+    selected candidate 64 PSF objects
+    fit ext:   1.151582 sec for 63 of 64 sources
+    fit psf:   0.476687 sec for 45 of 64 sources
+    chisq vs flux fit: 0.006806 +/- 0.227294
+    chisq vs flux fit term 0: 0.035698 +/- 0.200890
+    chisq vs flux fit term 1: 2.193609 +/- 0.292923
+    apresid: -0.018228 +/- 0.021941; keeping 29 of 64 psf stars
+    try model PS_MODEL_QGAUSS, ap-fit: -0.018228 +/- 0.021941 : sky bias: 0.000000
+    dSxo: mean: 0.003162, median: 0.001405, stdev: 0.013455, npts: 29
+    dSyo: mean: 0.000718, median: 0.002953, stdev: 0.012298, npts: 29
+    ndSx: mean: 0.045632, median: -0.045638, stdev: 1.276208, npts: 29
+    ndSy: mean: -0.089024, median: 0.223447, stdev: 1.087374, npts: 29
+    generate residuals for 29 objects: 0.270703 sec
+    select psf model: 1.917943 sec
+    psf model PS_MODEL_QGAUSS, ApResid: -0.018228 +/- 0.021941
+    no external sources supplied
+    built models for 100 objects: 0.232271 sec
+    measure ensemble of PSFs: 0.396416 sec
+    GrowthCurve : apLoss : 0.012242
+    measure aperture residuals for 64 objects (2 skipped, 0 failed, 34 invalid)
+    measure full-frame aperture residuals for 27 of 64 objects: 0.879843 sec
+    aperture residual: -0.012413 +/- 0.019795 : 0.000000 bias, 0.000000 skysat
+    measure magnitudes : 0.558418 sec for 100 objects (86 with apertures)
+    replace background flux : 0.056657 sec
+    creating moments plot
+    saving plot to firsttry.mnt.png
+    creating psf model plot
+    saving plot to firsttry.psf.png
+    creating ap mag - psf mag plot
+    saving plot to firsttry.dap.png
+    complete psphot run: 8.708983 sec
+ found 0 leaks at psphot
+
+'''I also tried this on the Mac in Heidelberg and get different results...:'''
+
+    Camera SDSS, format SDSS matches header.
+    Chip 0: 1 1
+    Cell 0: 1 1
+    Cell 0: 1 1
+    PHOTCODE is SDSS.i.21
+    System random seed value used  seed = 2726282019464764904 hex
+    image sky : mean 1311.806081 stdev 8.951074
+    background sky : min 1305.236572 mean 1311.805908 max 1372.219238 stdev 8.360644
+    subtracted background model: 0.268568 sec
+    built smoothed signficance image: 0.662050 sec
+    99 peaks: 0.081675 sec
+    96 sources, 96 moments, 3 failed: 0.094184 sec
+    found 0 satstar blend peaks, leaving 96 sources: 0.000078 sec
+    identified 8 blended objects: 0.007648 sec
+    psf clump  X,  Y: 1.645948, 1.765079
+    psf clump DX, DY: 0.056189, 0.045065
+    SN range (moments): 26.627832 - 1668.060669
+    SN range (peaks)  : 26.431179 - 1264.521973
+    rough classification: 0.012332 sec
+    selected candidate 64 PSF objects
+    fit ext:   0.840499 sec for 63 of 64 sources
+    fit psf:   0.416352 sec for 42 of 64 sources
+    chisq vs flux fit: 0.005232 +/- 0.371809
+    chisq vs flux fit term 0: -0.088947 +/- 0.187928
+    chisq vs flux fit term 1: 2.666661 +/- 0.175476
+    apresid: -0.037835 +/- 0.032469; keeping 26 of 64 psf stars
+    try model PS_MODEL_QGAUSS, ap-fit: -0.037835 +/- 0.032469 : sky bias: 0.000000
+    dSxo: mean: 0.001541, median: -0.001044, stdev: 0.014191, npts: 26
+    dSyo: mean: 0.001470, median: 0.001485, stdev: 0.011560, npts: 26
+    ndSx: mean: 0.125409, median: -0.191685, stdev: 1.329126, npts: 26
+    ndSy: mean: -0.105611, median: 0.018051, stdev: 1.013325, npts: 26
+    generate residuals for 26 objects: 0.283203 sec
+    select psf model: 1.554244 sec
+    psf model PS_MODEL_QGAUSS, ApResid: -0.037835 +/- 0.032469
+    no external sources supplied
+    built models for 96 objects: 0.170146 sec
+    measure ensemble of PSFs: 0.447365 sec
+    GrowthCurve : apLoss : 0.008937
+    measure aperture residuals for 64 objects (3 skipped, 0 failed, 29 invalid)
+    measure full-frame aperture residuals for 28 of 64 objects: 0.845621 sec
+    aperture residual: -0.012332 +/- 0.020148 : 0.000000 bias, 0.000000 skysat
+    measure magnitudes : 0.417724 sec for 96 objects (85 with apertures)
+    replace background flux : 0.027076 sec
+    skipping moments plot
+    skipping psf model plot
+    skipping ap-mag resid plot
+    complete psphot run: 5.548932 sec
+
+I thought I was using exactly the same settings (copied all the config files 1:1 from IfA to MPIA), but maybe not? E.g. why is it skipping the moments plot in HD, but not in HI? According to Gene, the different numbers are explicable by different random numbers (i.e. seeds). The plots missing would only make sense if kapa wasn't compiled properly, but we think it was.
+
+= Step 3: inserting results into dvo and inspecting them =
+
+psphot produces its output catalog in firsttry.cmf which we load into dvo:
+
+ addstar -D CAMERA sdss -accept-astrom firsttry.cmf
+
+-accept-astrom says that dvo should trust the WCS and ignore the fact that there are 0 astrometric stars reported in the header. Ohana commands like addstar require the -D CAMERA flag on the command line.
+
+Then start dvo by typing 'dvo' and plot things:
+
+ # Make a plot window
+ region 0 0 90 ait
+ cgrid
+ current style -x 2 -c black -pt 2 -lt 0 -lw 0.500000 -sz 1.000000
+ # The above line is the output from csgrid.  It's not something to type in.
+ # (May need to repeat both commands after adjusting size of kapa plot window)
+ 
+ # tell it where the catalogs are
+ catdir /data/ipp001.0/jester/stripe82/coadd/catdir
+ 
+ # make a new display for displaying images
+ images -n 1
+ 
+ # read a fits image into a buffer buf0
+ rd buf0 /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0192.fit
+ 
+ # display it with cuts given
+ tv buf0 1300 3000
+ # in the TV window, click left to pan, middle to zoom out, right to zoom in
+ 
+ # Tell IPP which region of the sky I'm interested in
+ skyregion 350 355 -2 0
+ # Extract individual measurements that exist in catdir for that region
+ mextract ra,dec,mag
+ # Show how many entries each vector has
+ vectors
+    N       name      size
+    0         ra        100
+    1        dec        100
+    2        mag        100
+ # convert ra,dec positions into display's x,y positions '''in place'''
+ coords buf0 -c ra dec
+ # and overlay the object positions on the display window
+ vload red ra dec
+ # To just plot the ra,dec on a coordinate grid (i.e., not an image), say
+ cplot ra dec
+
+= Step 4: warping and stacking =
+
+== Set up a grid of skycells to which the output will be warped ==
+
+DVO has Gene's tesselation of the sky built in. To create a table with the skycells of this tesselation, there's the skycells executable:
+
+ skycells 8 -scale .3 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ 
+
+The '8' tells it the tiling level, where 1 is coarse and 10 is fine. -scale gives the arcseconds per pixel. The skycells table is put into the directory specified by CATDIR.
+
+To look at the resulting tesselation, bring up dvo:
+
+ # Set catdir to the directory where the skycells live
+ catdir /data/ipp001.0/jester/stripe82/coadd/skycells/
+ # Plot them up, first set up the grid (args are ra dec radius projection)
+ region 352 -1 2 ait
+ images
+ # Overplot the outline of a fits image
+ imbox -c red /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0193.fit 
+ # Same for a list of images - first read list into a list variable
+ list myImlist -x "ls /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fit"
+ # Plot them up in a for loop
+ for i 0 $myImlist:n
+ >> imbox -c red $myImlist:$i
+ >> end
+ 
+ # The preceding was for individual fits files; if dvo already knows about images (via addstar), it works like this:
+ # set catdir to place where images are
+ catdir /data/ipp001.0/jester/stripe82/coadd/catdir/
+ # plot them up
+ images -c red
+ 
+ # To set up the next step, we need to know the id of the skycell. This can be done semi-interactively:
+ cursor
+ # then move the cursor into the skycell whose number you want to know and press any letter key,
+ # e.g. d, to store the ra,dec of the point where you press the key (q to quit, also stores things in q).
+ # $Rq and $Dq then store the ra,dec of that point, and you can use them in 'gimages' to get the skycell id:
+ gimages $Rq $Dq -region
+ 
+ output-> reading images (mode DVO_MODE_MEF)
+ output->  0 skycell.0484667 353.0672  -1.1307  1970/01/01,00:00:00     0  0 0.00  0.000 0.000 0.000    1
+
+skycell.0484667 is the ID we need.
+
+== Warp images to the skycell grid ==
+
+=== Setup: create a pixelless fits image containg the WCS of the skycell ===
+
+This uses a shell command:
+
+ dvoImageExtract skycell.0484667 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ -o /data/ipp001.0/jester/stripe82/coadd/run_ipp/skycell_0484667_warptemplate.fits
+
+to create the "warp template" fits file. It's a fits image with NAXIS=0, i.e., no pixels - just the header.
+
+=== Run psWarp ===
+
+You need to have the config files for the "simple" camera to run psWarp as follows:
+
+ pswarp -file ../input/fpC-001056-i1-0192.fit psWarp-001056-i1-0192 skycell_0484667_warptemplate.fits
+ pswarp -file ../input/fpC-001755-i1-0330.fit psWarp-001755-i1-0330 skycell_0484667_warptemplate.fits
+
+The 'output' parameter gives a rootname, so no .fits needed - it gets added automatically. The skycell needs to be specified with its full pathname.
+
+== Stack the images with ppStack ==
+
+To run ppStack, you need an input file that specifies input images, weights, masks, seeing; here's an [[Image(htdocs:/images/PpStack.mdc.txt)]] (the .txt is just there because the wiki judges files by its name and doesn't know .mdc files). You then simply say:
+
+ ppStack ppStack.mdc ppStack-run1
+
+where ppStack-run1 is the output rootname.
+
+= Step 5: run psphot on the stacked image =
+
+Initially, this barfed with some config file issue, but Paul fixed it. Now it runs with [[Image(htdocs:/images/Photstack-run1.out.txt)]]
+
+ psphot -file ppStack-run1.fits photstack-run1
+
+= Step 6: inject stacked psphot results into dvo =
+
+= Step 7 (really chapter 2): run everything in automated fashion in pantasks =
+
+== Preliminaries: set up a database in ippMon ==
+
+The db administrator uses ippMonitor/dbadmin to create a new "project" in the mySQL database (this isolates different reduction runs from each other; I get the impression that it's useful to really have one project per run, as the easiest [and perhaps only?] way to start over with something is to delete all the tables in the project and start from scratch).
+
+=== Administrator things ===
+
+If the database is being set up for the very first time at all for IPP use, the admin need sto say (e.g.)
+
+ dbadmin init (dbserver) (dbname) (dbpassword)
+ dbadmin init aida77 ipp ********
+
+This creates an administrative database ippadmin which keeps track of everything else and is just required once per IPP installation.
+
+To set up a new project database, then say
+
+ dbadmin project (dbserver) (dbuser) (dbname)
+ dbadmin project aida77 ipp jester_s82
+
+confirming with the password set above. This creates a project database jester_s82 that keeps track of all files in that project.
+
+There's also something about creating a www user:
+
+ dbadmin user (dbserver) (dbuser) (username) (password)
+
+but I'm still fighting with the web server. NB: in our case, the php script install location is
+
+ /srv/www/htdocs/ippMonitor
+
+=== User things ===
+
+The server, ipp-dbname, password and project are specified in ~/.ipprc and/or site.config. I have in site.config:
+
+ ### Database configuration
+ DBSERVER       STR     aida77                  # Database host name (for psDBInit)
+ DBUSER         STR     ipp                     # Database user name (for psDBInit)
+ DBPASSWORD     STR     *******                 # Database password (for psDBInit)
+
+and in ~/.ipprc:
+
+ DBNAME         STR     jester_s82              # Database name (for psDBInit)
+
+I as user then use pxadmin to create a new database, or delete or recreate it (make sure that DBNAME is pointing to the database you actually want to work on!)
+
+ pxadmin -create
+
+'''There should proabably be something like an "IPP administrator's guide" (or maybe there is already one) that spells out how mySQL should be set up, what the best way is to make sure that different users don't interfere with each other's tables and projects (e.g. by having multiple mySQL users, if possible), etc.'''
+
+To start a whole reduction afresh, at the moment the fastest/only way is
+
+ pxadmin -recreate
+
+This will delete EVERYTHING in your database DBNAME...
+
+== Injecting and registering frames in ippMon ==
+
+Now we need to tell ippMon about our images. This has been covered in [[Image(htdocs:/images/IPP-1-1 tutorial.pdf)]]] and also in [http://kiawe.ifa.hawaii.edu/eugene/software/ipp-workshop.pdf Gene and Paul et al's giant 2-day presentation on IPP from the Baltimore workshop] (slide 228).
+
+Injection means telling IPP about the files
+
+=== Adding a static bad pixel mask to the db ===
+
+I've written a [[Image(htdocs:/images/RegisterBPmasks.txt)]]  path://S82/sdssBPmask/registerBPmasks.pl that runs the injection of the bad-pixel masks, first defining a ''detrend run'' for each filter, then injecting all the [[Image(htdocs:/images/Sdss_staticbadpixmasks.txt)]] I created from the [[Image(htdocs:/images/OpBC-51392.txt)]]. The bad pixel masks only work with the mosaiced camera below - in fact, they were the motivation for introducing the mosaiced SDSS camera.
+
+'''After pxadmin recreate, the bad-pixel masks need to be injected ''before'' the image files, so they are there and can be used!'''
+
+=== Injection with the "single chip = FPA" config ===
+
+This is in 
+
+ ~jester/IPP/ippconfig/sdss/camera.config 
+
+The injection is simple here:
+ ipp_serial_inject.pl --telescope sdss --camera sdss \
+ --workdir path://S82/run_ipp --dbname jester \
+ --reduction PROCESSED /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fits
+
+The "reduction class" PROCESSED selects a given set of reduction steps that is defined in the ppImage.config file. In the sdssmosaic/ppImage.config, I renamed this class into SDSSphotonly, which runs psphot but not psastro, to distinguish it from SDSSphotast, which runs both.
+
+=== Injection with the "six chips = mosaiced FPA" config ===
+
+This is in 
+
+ ~jester/IPP/ippconfig/sdssmosaic/
+
+and the camera name is SDSSmosaic. Whether an SDSS image is recognized as SDSSmosaic or the single-chip SDSS format depends on which of the lines in my ~/.ipprc is commented out - the format.config for both files look identical, since the files ''are'' identical.
+
+The injection is more involved, since the injection needs to know about the grouping. There are two possibilities: ipp_serial_inject_mosaic.pl and ipp_serial_inject_split.pl
+
+The former needs to be modified to know about the structure of SDSS. It then needs to be called once per group of 6 files that are part of the same "exposure".
+
+The latter needs to have the input files put into subdirectories, with one dir per "exposure". You call it with a glob for all the directories [I think]. 
+
+Since I need to write a script either way, either to copy to directories or to call with a suitable glob per exposure, I'll take the latter route so we don't need to change ipp_serial_inject_mosaic.pl.
+
+To inject all the files in a given directory $D/stripe82/coadd/input/i/1056-0192/, use this command:
+
+ ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \
+ --workdir path://S82/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82CATDIR \
+ $D/data/SDSS/stripe82/coadd/input/i/1056-0192/
+
+tess_id and dvodb are new in 2.5. The tesselation_name and dvodb_name can be explicit paths to catdirs above, but also abstract names defined in site.config
+
+SDSSphotonly corresponds to the PROCESSED reduction class (now PPIMAGE_OP_SOFT) defined in sdss[mosaic]/camara.config using the recipes in sdss[mosaic]/ppImage.config. The other new reduction class is SDSSphotast, which includes astrometry.
+
+For the initial testing, I want to isolate any possible astrometry problems from photometry problems, so I'll continue to work with SDSSphotonly. However, I wanted to make sure that phot & ast both run through, so that I can rectify any mechanical problems while still at the IfA, and then worry only about functional problems later.
+
+=== Registering ===
+
+"Registering" means getting them ingested into ippMon: 
+
+ # Start pantasks
+ pantasks
+ # Inside pantasks, load predefined procuedures
+ module pantasks.pro
+ module.tasks
+ # Add host names for parallel processing
+ controller host add ipp000
+ # etc. for more hosts.
+ # Now by default, all analysis steps are on by default, but we want 
+ # to go through them step by step to make sure everything's working.
+ # Hence we turn 'all' off, and then turn things on selectively.
+ # Also, at this stage, IPP doesn't know by itself that fpC*.fits images
+ # from SDSS don't need detrending, so we need to tell IPP by hand after the
+ # register step.
+ all.off
+ register.on
+ run
+
+You can also save these steps in a macro, by puttin the commands in a file, wrapping them in "macro myMacro" and "end", and then reading the file in pantasks with the "input" command. In pantasks, one can then call the macro by just typing myMacro. On the IfA cluster, my macro that does these things is called thus:
+
+ input /home/panstarrs/jester/IPP/macros/inject.pro
+ injectmine
+
+Note that it is possible to 'controller add' the same host multiple times, and it will get multiple shares of the load.
+
+== Processing things with the pipeline ==
+
+All the reduction steps are just switches in the pantasks.pro module; they can be run simply by turning them on, e.g.
+
+ chip.on
+ camera.on
+
+runs psphot.
+
+The next step is warping, which requires some extra command-line work (but maybe there's a way to have IPP do this, too? There has to be one, eventually...)
+
+== Warping ==
+
+=== Defining warp runs ===
+
+In IPP 2.5, there's a new feature where you specify the skycell tessellation id already in the [wiki:#Injection_with_the_.22six_chips_.3D_mosaiced_FPA.22_config injection step (see above)]. You then just need to turn on the warping step, and it figures out what to warp where by itself, and does it.
+
+=== Run them ===
+
+Then
+
+ warp.on
+ run
+
+will run the actual warping. To see what's going on, say
+
+ status
+
+for a general job count, and
+
+ status -taskstats
+
+to see statistics on execution times.
+
+== Stacking ==
+
+Now that everything is warped, we could stack it. For this, we need to define a 'stack run' with stacktool.
+
+ stacktool -definerun -workdir path://S82/run_ipp -skycell_id skycell.0472823 -warp_id 1 -warp_id 2 -warp_id 5 -tess_id Skycells
+
+This tells IPP to stack things in the skycell given and include the pictures from the warp runs listed. The tess_id is defined in the TESSELLATIONS METADATA in camera.config, and points to the same path that I used to set up the tessellation [wiki:#Set_up_a_grid_of_skycells_to_which_the_output_will_be_warped above]
+
+At the moment, you need to figure out yourself which skycells and warp_ids to use. I do this:
+
+ # First, connect to the database
+ mysql -h alala -u ipp -p jester
+ [enter password]
+ # then run the following query
+
+ select distinct skycell_id, warp_id from warpSkyCellMap 
+ where skycell_id in (select skycell_id from warpSkyCellMap 
+    group by skycell_id having count(distinct warp_id) > 1)
+ order by skycell_id, warp_id;
+
+The inner query shows you how many warp runs overlap a given skycell:
+
+ select skycell_id, count(distinct warp_id) as N_warp, count(*) as N_img 
+ from warpSkyCellMap 
+ group by skycell_id having N_img  > 1 
+ order by skycell_id ;
+
+Conversely, you can find out how many skycells each of your images was split into:
+
+ select cam_id, class_id, warp_id,  count(distinct skycell_id) 
+ from warpSkyCellMap 
+ group by class_id, warp_id, cam_id 
+ order by cam_id, class_id, warp_id;
+
+If something fails, the way to get IPP to stop trying is
+
+ stacktool -updaterun -stack_id 2 -state stop
+
+== Diffing == 
+
+In principle, we could also define a diff run with difftool, but since I'm not interested in image subtraction myself, at least not initially (sorting out brown dwarfs via proper motion may change this in the future), I am skipping this step. 
+
+However, the SDSS supernova survey has been preparing high-quality subtractions for this same data set, so it would be a very good comparison set, too. Gajus Miknaitis at Fermilab knows about the SDSS set, and I'm happy to help establishing contacts if necessary.
+
+== Running pshot on the stack ==
+
+Don't know how that works in the pipeline yet.
+
+== Loading stacked psphot outputs into dvo ==
+
+Should work as above, in principle.
+
+= Log 2008-05-28 =
+
+I don't know where I'm at on aida77 so I'm starting afresh.
+
+== Creating a database ==
+
+Start with fresh jester_s82 reduction database:
+ pxadmin -recreate -dbname jester_s82
+
+or create a new one for a new run:
+ ippadmin project aida77 ipp jester_s82_20080912
+ pxadmin -create -dbname jester_s82_20080912
+
+== Injecting == 
+
+Normally I'd want to inject the bad pixel masks as described above, but I'm skipping that for now. Instead, go straight to injection, for now just with all the fields from 2 runs to see that it works:
+
+ ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \
+ --workdir path://S82TEST/run_ipp --dbname jester_s82_20080912 --tess_id S82CELLS --dvodb S82TESTCATDIR  \
+ /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/
+
+or for everything:
+
+ ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \
+ --workdir path://S82PROD/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82PRODCATDIR  \
+ /IPP/data/SDSS/stripe82/coadd/input/?/*/
+
+The symbolic names like S82TEST are defined in <code>~/.ipprc</code> or <code>/IPP/site.config</code> (or another site config file pointed to by your <code>~/.ipprc</code>).
+
+=== With different recipes ===
+
+At some point, IPP allowed using recipes also for psphot, which are in sdssmosaic/psphot.config and sdssmosaic/reductionClasses.mdc For these, use things like
+
+ pxadmin -recreate -dbname jester_s82_20081217_ap3
+
+ ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic \
+ --reduction SDSSphotast_ap \
+ --workdir path://S82AP3_20081217 \
+ --dbname jester_s82_20081217_ap3 \
+ --tess_id S82CELLS \
+ --dvodb S82AP3CAT_20081217  \
+ /IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/
+
+This recipe turns on both aperture corrections and spatially variable PSF fits.
+
+== Pantasksing ==
+
+Then start jobs in pantasks:
+
+ pantasks
+ # If it's not in your ~/.pantasksrc
+ module pantasks.pro
+ module.tasks
+
+ # There's also a file with the sort of commands that I'm describing here
+ # in /home/jester/s82.pro which you can load with
+ input /home/jester/s82.pro
+ 
+ # Set database if different from DBNAME in ~/.ipprc
+ add.database jester_s82_20080912
+ 
+ # Turn all reduction steps off, except for register, chip and camera:
+ all.off
+ register.on
+ chip.on
+ camera.on
+ 
+ # Add some CPUs for the task scheduler to use
+ # - need to be able to ssh without passwords to the host given
+ # (see /IPP/README)
+ controller host add aida77 -threads 2
+ 
+ # This command can be issued multiple times for adding more
+ # CPUs on the same machine, or more machines.
+ # The -threads 2 is optional and allows multi-threading,
+ # which should be the fastest option.
+ # aida77 (and aida78 etc.) have 2 dual-core CPUs. Hence:
+ # Without the -threads, use at most 3 copies of aida77
+ # (fewer if more than one person are running pantasks!)
+ # With the threads, use at most 2 copies of aida77.
+ 
+ # Now start the processing
+ run
+ 
+ # say 'stop' to stop launching new jobs, or 'halt' to
+ # stop processing altogether (and being flooded by error messages)
+ 
+ # To see what's going on:
+ controller status
+ # lists active and pending jobs and the status of the hosts
+ 
+ status
+ # lists pending jobs and statistics of success/failures
+ 
+ status -taskstats
+ # includes information on how long each job ran
+
+NB: with
+
+ $VERBOSE = 2
+
+in pantasks, you see the commands as they're issued, which is useful for diagnosing problems. You can also simply re-run the command that's announced as failing, or look at the log file indicated. Normally, to re-run without generating further error messages, you need to remove the <code>--log path://datata</code> argument to see the output (which otherwise goes into a log file), and add <code>--no-update</code> so that the database isn't affected by the manual repeat.
+
+Once the camera-stage processing is finished, there's a dvo catalog with the outputs in S82PRODCATDIR. This needs to be sorted:
+ cd `ipp_datapath.pl path://S82PRODCATDIR`
+ addstar -D CATDIR . -resort
+ relphot -D CATDIR . -averages
+
+The resorted version is available for download at [http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz] - a tarfile of about 250 MB that expands into a directory of about 500 MB. I have asked our computer people also to make available all the IPP outputs directly; however, those are 250 GB (no compression) so they need to figure out a way to host those. The input fpC*.fits images from SDSS are another ~50 GB. Here is the [[Image(htdocs:/images/README_IPPstripe82_20080815.txt)]].tgz