Summary
Simtest: is a process for testing the IPP with simulated data. The simulated data is produced and injected into an SQL database by ipp_simulation_data.pl It can then be carried through the following ipp stages:
- Registration
- Detrend construction
- Chip processing
- Camera processing
- Warp to skycell
- Difference
- Stack
Annotated simtest walkthrough
Configuration:
- Check ~/.ipprc (or $PS_SITE) for:
- DATAPATH for SIMTEST, SIMWORK
- the directory pointed to by SIMTEST is where your RAW simtest data will be created and stored. You can refer to it in command -line arguments as path://SIMTEST/
- the SIMWORK directory is where the intermediate and final output files and all log files will be placed. Refer to it as path://SIMWORK/.
- (The variable name used, e.g. SIMTEST or SIMWORK, is unimportant. Just be consistent in the program calls below)
- DBSERVER, DBNAME, DBUSER, DBPASSWORD
- these set the parameters used by programs to access the SQL database
- DATAPATH for SIMTEST, SIMWORK
- Check ~/.ptolemyrc for:
- CONFDIR
- this is the directory containing your configuration files.
- CONFDIR
Initiating a database.
- There are three ways:
1) If you don't care to use the ippMonitor webpage, you can create an empty database in mysql
mysql
mysql> CREATE DATABASE mydatabase; mysql> exit
then initiate the new database
pxadmin -create -dbname mydatabase
2) If you are restarting an existing database project you can delete and rebuild an empty simtest database:
pxadmin -recreate -dbname mydatabase
3) You can use dbadmin to create an empty database that will be recognized by ippMonitor (note: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere. Call dbadmin with no options for syntax help.)
cd /data/po02.0/ipp/ipp-2.1/ippMonitor dbadmin
USAGE: dbadmin (options)
dbadmin init (dbserver) (dbuser) (dbpassword)
creates admin tables, basic db user -- requires root password
dbadmin client (dbserver) (dbuser) (dbpassword) (client)
add client name or regex
dbadmin user (dbserver) (dbuser) (username) (password)
create a new www user and password
dbadmin project (dbserver) (dbuser) (dbname)
create a new ipp project (database)
dbadmin project alala ipp mydatabase
then initiate the new database
pxadmin -create -dbname mydatabase
Make sim data
(and inject it into the database)
ipp_simulation_data.pl --name test --dbname mydatabase --camera SIMTEST --telescope SimScope --workdir path://SIMWORK/ --path path://SIMTEST/
wait until it finishes ...
Start pantasks
pantasks is the ipp parallel process manager for distributed computing across multiple nodes. start up pantasks from a second window
pantasks
Welcome to pantasks - parallel task scheduler
load some pantasks commands
pantasks: module pantasks.pro
or, if you have a modified pantasks.pro file
pantasks: input /home/username/pantasks/pantasks.pro
initialize controller connections, defined in your pantasks.pro file. (you must already have set up passphraseless ssh to any machine you wish to use as a controller)
pantasks: init.cluster.po starting controller connection Connected HostID: 0 HostID: 1 HostID: 2 HostID: 3 HostID: 4
To add an individual controller host by hand:
pantasks: controller host add myhost
If you are using a database other than the one declared in your .ipprc file, use the DB variable. DB holds a list of databases. $DB:0 is the active database. The default database defined in your .ipprc file will only be used if $DB:n is 0.
pantasks: list DB -add mydatabase pantasks: echo $DB:0 mydatabase
Registration
The newly injected simulated data is in the database, but we don't know what each image is yet, so we register all the new exposures in the database, getting their vital statistics and sorting them appropriately. In pantasks, start up the tasks list, turn off some switches and check the status
pantasks: module.tasks pantasks: chip.off pantasks: camera.off pantasks: warp.off pantasks: stack.off pantasks: detrend.off pantasks: diff.off pantasks: status
Scheduler is stopped Controller is stopped
Task StatusAV Name Njobs Ngood Nfail Ntime Command ++ register.imfile.load 0 0 0 0 (dynamic) ++ register.imfile.run 0 0 0 0 (dynamic) ++ register.exp.load 0 0 0 0 (dynamic) ++ register.exp.run 0 0 0 0 (dynamic) -+ chip.imfile.load 0 0 0 0 (dynamic) -+ chip.imfile.run 0 0 0 0 (dynamic) -+ camera.exp.load 0 0 0 0 (dynamic) -+ camera.exp.run 0 0 0 0 (dynamic) -+ dettool.process.load 0 0 0 0 (dynamic) -+ dettool.process.run 0 0 0 0 (dynamic) -+ dettool.processexp.load 0 0 0 0 (dynamic) -+ dettool.processexp.run 0 0 0 0 (dynamic) -+ dettool.stack.load 0 0 0 0 (dynamic) -+ dettool.stack.run 0 0 0 0 (dynamic) -+ dettool.norm.load 0 0 0 0 (dynamic) -+ dettool.norm.run 0 0 0 0 (dynamic) -+ dettool.normexp.load 0 0 0 0 (dynamic) -+ dettool.normexp.run 0 0 0 0 (dynamic) -+ dettool.normstat.load 0 0 0 0 (dynamic) -+ dettool.normstat.run 0 0 0 0 (dynamic) -+ dettool.resid.load 0 0 0 0 (dynamic) -+ dettool.resid.run 0 0 0 0 (dynamic) -+ dettool.residexp.load 0 0 0 0 (dynamic) -+ dettool.residexp.run 0 0 0 0 (dynamic) -+ dettool.reject.load 0 0 0 0 (dynamic) -+ dettool.reject.run 0 0 0 0 (dynamic) -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) no defined jobs
Start the registration step
pantasks: run
You can check the progress of the registration with status calls or on the web monitor: http://kiawe.ifa.hawaii.edu/ippMonitor
When all your images have finished registration you can halt the register.load processes, which are no longer needed. (or they can continue to run in the background.)
pantasks: halt
Detrend construction
Flip switches for the detrend step
pantasks: register.off pantasks: detrend.on
Back in a base shell, outside of pantasks, run the detrend tool. This creates a detRun table instance in the database, from which pantasks can get the info it needs to direct the detrend construction
dettool -definebyquery -det_type BIAS -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type BIAS
You should now be able to find an entry for this detrend run on the ippMonitor web page or with a MySQL query on the detRun table. When you see it there, start the pantasks scheduler in your pantasks window, window 2.
pantasks: run
When you see the bias in the ippMonitor under Detrend Frames or in MySQL under detFrames, then go back to window 1 for a sequence of dettools for DARKs, SHUTTERs, and FLATs. After running each dettool command from the command line, you should be able to find the run on the ippMonitor or in MySQL.
dettool -definebyquery -det_type DARK -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type DARK
Wait for the dark to finish before starting to make the shutter
dettool -definebyquery -det_type SHUTTER -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
Wait for the shutter to finish before starting the flats (but both flats can be done at once)
dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter i -select_exp_type FLAT -select_filter i
Check on the status in pantasks
pantasks: status
Scheduler is running Controller is running
Task Status
AV Name Njobs Ngood Nfail Ntime Command -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ register.imfile.load 31 31 0 0 regtool -+ register.imfile.run 74 74 0 0 register_imfile.pl -+ register.exp.load 36 36 0 0 regtool -+ register.exp.run 74 74 0 0 register_exp.pl ++ dettool.process.load 314 314 0 0 dettool ++ dettool.process.run 75 75 0 0 detrend_process_imfile.pl ++ dettool.processexp.load 314 314 0 0 dettool ++ dettool.processexp.run 75 75 0 0 detrend_process_exp.pl ++ dettool.stack.load 314 314 0 0 dettool ++ dettool.stack.run 5 5 0 0 detrend_stack.pl ++ dettool.norm.load 314 314 0 0 dettool ++ dettool.norm.run 5 5 0 0 detrend_norm_apply.pl ++ dettool.normexp.load 314 314 0 0 dettool ++ dettool.normexp.run 5 5 0 0 detrend_norm_exp.pl ++ dettool.normstat.load 314 314 0 0 dettool ++ dettool.normstat.run 5 5 0 0 detrend_norm_calc.pl ++ dettool.resid.load 314 314 0 0 dettool ++ dettool.resid.run 75 75 0 0 detrend_resid.pl ++ dettool.residexp.load 314 314 0 0 dettool ++ dettool.residexp.run 75 75 0 0 detrend_reject_imfile.pl ++ dettool.reject.load 312 312 0 0 dettool ++ dettool.reject.run 5 5 0 0 detrend_reject_exp.pl
no defined jobs
Chip and Camera
When detrend creation is done, start the chip and camera processing. This takes all the science images and does the detrending (dark sub, bias sub, shutter correction, flat fielding)
pantasks: stop pantasks: detrend.off pantasks: chip.on pantasks: camera.on pantasks: run
...
pantasks: stop
Warp
Next we prepare for warping the science images into common skycells. i.e. putting them on a common tessellated tangent plane projection
If the skycells have not been defined, then the skycells program will make them. This creates a directory with catalogs of skycell definitions.
skycells -D CATDIR /data/ipp002.0/price/skycells 8 -scale 0.2
no data in db /data/ipp002.0/price/skycells/Photcodes.dat directory /data/ipp002.0/price/skycells does not exist, creating...
IMPORTANT: be sure to point to this directory in your camera.config file with, e.g.:
TESSELLATIONS METADATA
# Tessellation name with corresponding DVO catdir Skycells STR path://SIMTEST/skycells
END
As of ipp-2.1 we still have to do these final stages more or less by hand. In later releases there will be a perl script to craft the sql queries that define a filter set for a given skycell, etc.
Here is an example sql query to get the cam_id's of all the r-band exposures:
mysql -h alala -u ipp -pipp simtest
mysql> select camProcessedExp.cam_id, rawExp.exp_tag from camProcessedExp JOIN chipProcessedExp USING(chip_id) JOIN rawExp USING(exp_tag) where rawExp.filter = 'r'; +--------+-------------+ | cam_id | exp_tag | +--------+-------------+ | 1 | test0060.61 | | 2 | test0063.64 | | 3 | test0062.63 | | 4 | test0064.65 | | 5 | test0065.66 | | 8 | test0061.62 | | 13 | test0066.67 | +--------+-------------+ 7 rows in set (0.00 sec)
Now we use warptool (by hand at the moment) to define the warp runs. There is a single run for each image being warped. Warptool assigns a sequential warp_id each time it is called, but as yet it knows nothing about the data being warped. In this example we are defining 7 warp runs that will later be assigned to the 7 r-band images listed above:
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 1 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:24:19Z
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 2 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:10Z
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 3 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:12Z
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 4 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:13Z
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 5 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:14Z
warptool -definerun -mode warp -workdir path://SIMTEST/
warp_id S64 6 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:16Z
warptool -definerun -mode warp -workdir path://SIMWORK/
warp_id S64 7 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:17Z
Now we can use warptool to add the input exposure to each run. The input exposures are identified by their cam_id's, which we collected in mysql above.
warptool -addinputexp -warp_id 1 -cam_id 1 warptool -addinputexp -warp_id 2 -cam_id 2 warptool -addinputexp -warp_id 3 -cam_id 3 warptool -addinputexp -warp_id 4 -cam_id 4 warptool -addinputexp -warp_id 5 -cam_id 5 warptool -addinputexp -warp_id 6 -cam_id 8 warptool -addinputexp -warp_id 7 -cam_id 13
Next, we set the state of each warp run from 'stop' (the default) to 'run' so that pantasks can process them.
warptool -updaterun -warp_id 1 -state run warptool -updaterun -warp_id 2 -state run warptool -updaterun -warp_id 3 -state run warptool -updaterun -warp_id 4 -state run warptool -updaterun -warp_id 5 -state run warptool -updaterun -warp_id 6 -state run warptool -updaterun -warp_id 7 -state run
Finally, go back to your pantasks window and start it up:
pantasks: warp.on pantasks: run
...
pantasks: stop
Difference
The process is similar for difftool and stacktool. We run difftool once to define a run:
difftool -definerun -workdir path://SIMTEST/ -skycell_id skycell.0743078 -tess_id skycells
diff_id S64 1 state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T02:12:53Z skycell_id STR skycell.0743078 tess_id STR skycells
Run difftool a second (and third) time to add the skycells we are differencing after running warptool, the warped image files are identified by the warp_id that was assigned by warptool above):
difftool -addinputskyfile -diff_id 1 -warp_id 1 -skycell_id skycell.0743078 -tess_id skycells /data/ipp002.0/price>difftool -addinputskyfile -diff_id 1 -warp_id 2 -skycell_id skycell.0743078 -tess_id skycells -template
And a final time to set the state to 'run':
/data/ipp002.0/price>difftool -updaterun -diff_id 1 -state run
Then go back to pantasks and fire it up:
pantasks: warp.off pantasks: diff.on pantasks: run
...
pantasks: stop
Stack
Same deal for stacktool. When you define the run, give it a list of all the warp_id's that you want to stack and add.
stacktool -definerun -skycell_id skycell.0743078 -tess_id skycells -workdir path://SIMTEST/ -warp_id 1 -warp_id 2 -warp_id 3 -warp_id 4 -warp_id 5 -warp_id 6 -warp_id 7
Maybe have to set the state here?
Then tell pantasks to go
pantasks: diff.off pantasks: stack.on pantasks: run
...
pantasks: stop
Lather, rinse, repeat.
