Summary
Simtest: is a process for testing the IPP with simulated data. The simulated data is produced and injected into an SQL database by ipp_simulation_data.pl It can then be carried through the following ipp stages:
- Registration
- Detrend construction
- Chip processing
- Camera processing
- Warp to skycell
- Difference
- Stack
Updated for ipp-2.4 / ipp-2.5 by S.Rodney
Annotated simtest walkthrough
Configuration:
- Check ~/.ipprc (or $PS_SITE)
- SITE and SYSTEM point at valid site.config and system.config files
- PATH should contain your config directory
- site.config: DATAPATH for SIMTEST, SIMWORK
- the directory pointed to by SIMTEST is where your RAW simtest data will be created and stored. You can refer to it in command -line arguments as path://SIMTEST/
- the SIMWORK directory is where the intermediate and final output files and all log files will be placed. Refer to it as path://SIMWORK/.
- (The variable name used, e.g. SIMTEST or SIMWORK, is unimportant. Just be consistent in the program calls below)
- DBSERVER, DBNAME, DBUSER, DBPASSWORD
- these set the parameters used by programs to access the SQL database
- Check ~/.ptolemyrc for:
- CONFDIR
- this is the directory containing your configuration files.
- CONFDIR
Initiating a database.
- There are three ways:
1) If you don't care to use the ippMonitor webpage, you can create an empty database in mysql
mysql
mysql> CREATE DATABASE mydatabase; mysql> exit
then initiate the new database
pxadmin -create -dbname mydatabase
2) If you are restarting an existing database project you can delete and rebuild an empty simtest database:
pxadmin -recreate -dbname mydatabase
3) You can use dbadmin to create an empty database that will be recognized by ippMonitor (note: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere. Call dbadmin with no options for syntax help.)
cd /data/po02.0/ipp/ipp-2.1/ippMonitor dbadmin
USAGE: dbadmin (options)
dbadmin init (dbserver) (dbuser) (dbpassword)
creates admin tables, basic db user -- requires root password
dbadmin client (dbserver) (dbuser) (dbpassword) (client)
add client name or regex
dbadmin user (dbserver) (dbuser) (username) (password)
create a new www user and password
dbadmin project (dbserver) (dbuser) (dbname)
create a new ipp project (database)
dbadmin project alala ipp mydatabase
then initiate the new database
pxadmin -create -dbname mydatabase
Make sim data
(and inject it into the database)
ipp_simulation_data.pl --name test --dbname mydatabase --camera SIMTEST --telescope SimScope --workdir path://SIMWORK/ --path path://SIMTEST/
wait until it finishes ...
Start pantasks
pantasks is the ipp parallel process manager for distributed computing across multiple nodes. start up pantasks from a second window
pantasks
Welcome to pantasks - parallel task scheduler
load some pantasks commands
pantasks: module pantasks.pro
or, if you have a modified pantasks.pro file
pantasks: input /home/username/pantasks/pantasks.pro
Add some CPU 'controllers' to carry out the processing steps. Each controller you add will start an instance of the 'pclient' program, which receives processes from the pantasks 'pcontroller.' To add an individual controller host by hand:
pantasks: controller host add myhost
NOTE: You must have passphraseless ssh authentication set up from your pcontroller machine (the one running pantasks) to your pclient machines (the ones added with 'controller host add'). You may want to set up a module in your pantasks.pro file to quickly add a set of controllers with one command.
If you are using a database other than the one declared in your site.config file, use the DB variable. DB holds a list of databases. $DB:0 is the active database. The default database defined in your site.config file will only be used if $DB:n is 0.
pantasks: list DB -add mydatabase pantasks: echo $DB:0 mydatabase
The pantasks.pro module should contain a function called add.database to do this as well.
pantasks: add.database mydatabase
Registration
The newly injected simulated data is in the database, but we don't know what each image is yet, so we register all the new exposures in the database, getting their vital statistics and sorting them appropriately. In pantasks, start up the tasks list, turn off some switches and check the status
pantasks: module.tasks pantasks: all.off pantasks: register.on pantasks: status Task Status
AV Name Njobs Ngood Nfail Ntime Command -+ detrend.process.load 0 0 0 0 (dynamic) -+ detrend.process.run 0 0 0 0 (dynamic) -+ detrend.processexp.load 0 0 0 0 (dynamic) -+ detrend.processexp.run 0 0 0 0 (dynamic) -+ detrend.stack.load 0 0 0 0 (dynamic) -+ detrend.stack.run 0 0 0 0 (dynamic) -+ detrend.norm.load 0 0 0 0 (dynamic) -+ detrend.norm.run 0 0 0 0 (dynamic) -+ detrend.normexp.load 0 0 0 0 (dynamic) -+ detrend.normexp.run 0 0 0 0 (dynamic) -+ detrend.normstat.load 0 0 0 0 (dynamic) -+ detrend.normstat.run 0 0 0 0 (dynamic) -+ detrend.resid.load 0 0 0 0 (dynamic) -+ detrend.resid.run 0 0 0 0 (dynamic) -+ detrend.residexp.load 0 0 0 0 (dynamic) -+ detrend.residexp.run 0 0 0 0 (dynamic) -+ detrend.reject.load 0 0 0 0 (dynamic) -+ detrend.reject.run 0 0 0 0 (dynamic) -+ chip.imfile.load 0 0 0 0 (dynamic) -+ chip.imfile.run 0 0 0 0 (dynamic) -+ camera.exp.load 0 0 0 0 (dynamic) -+ camera.exp.run 0 0 0 0 (dynamic) -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) ++ register.imfile.load 0 0 0 0 (dynamic) ++ register.imfile.run 0 0 0 0 (dynamic) ++ register.exp.load 0 0 0 0 (dynamic) ++ register.exp.run 0 0 0 0 (dynamic)
Start the registration step
pantasks: run
You can check the progress of the registration with status calls or on the web monitor: http://kiawe.ifa.hawaii.edu/ippMonitor
If you are watching the processes on your controller hosts (e.g. using 'top') then you should see pclient processes. The pantasks process itself is called pcontrol.
When all your images have finished registration you can halt the register.load processes, which are no longer needed. (or they can continue to run in the background.) You can turn off the registration switches.
pantasks: stop pantasks: register.off
NOTE: using 'halt' instead of 'stop' will stop the processes without harvesting jobs.
Detrend construction
Flip switches for the detrend step
pantasks: detrend.on
Back in a base shell, outside of pantasks, run the detrend tool. This creates a detRun table instance in the database, from which pantasks can get the info it needs to direct the detrend construction
dettool -dbname mydatabase -definebyquery -det_type BIAS -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type BIAS
You should now be able to find an entry for this detrend run on the ippMonitor web page or with a MySQL query on the detRun table. When you see it there, start the pantasks scheduler in your pantasks window, window 2.
pantasks: run
When you see the bias in the ippMonitor under Detrend Frames or in MySQL under detFrames, then go back to window 1 for a sequence of dettools for DARKs, SHUTTERs, and FLATs. After running each dettool command from the command line, you should be able to find the run on the ippMonitor or in MySQL.
dettool -dbname mydatabase -definebyquery -det_type DARK -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type DARK
Wait for the dark to finish before starting to make the shutter
dettool -dbname mydatabase -definebyquery -det_type SHUTTER -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
Wait for the shutter to finish before starting the flats (but both flats can be done at once)
dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter i -select_exp_type FLAT -select_filter i
When detrend creation is done, you can stop and switch off the detrend processing.
pantasks: stop pantasks: detrend.off
Chip and Camera
Start the chip and camera processing. This takes all the science images and does the detrending (dark sub, bias sub, shutter correction, flat fielding)
<pre>
pantasks: chip.on pantasks: camera.on pantasks: run
...
pantasks: stop
</pre>
Warp
Next we prepare for warping the science images into common skycells. i.e. putting them on a common tessellated tangent plane projection
If the skycells have not been defined, then the skycells program will make them. This creates a directory with catalogs of skycell definitions.
skycells -D CATDIR ./skycells 8 -scale 0.2
no data in db ./skycells/Photcodes.dat directory ./skycells does not exist, creating...
IMPORTANT: be sure to point to this directory in your site.config file with, e.g.:
TESSELLATIONS METADATA
# Tessellation name with corresponding DVO catdir Skycells STR path://SIMTEST/skycells
END
Now we should be able to go back to the pantasks window and start warping:
<pre>
pantasks: warp.on pantasks: run
...
pantasks: stop
</pre>
The warped images will be stored in path://SIMWORK/tess_skycells/sky_skycell.<skycellnum>/ as '*.wrp[1-7].fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats'). The warping will automatically calculate the appropriate "<skycellnum>" for the given tessellation and location of image on the sky.
Difference
<pre>
pantasks: warp.off pantasks: diff.on pantasks: run
...
pantasks: stop
</pre>
These difference image will be stored in path://SIMWORK/tess_skycells/sky_skycell.0743078/ as '*.dif1.fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats').
Stack
<pre>
pantasks: diff.off pantasks: stack.on pantasks: run
...
pantasks: stop
</pre>
Lather, rinse, repeat.
