Changes between Initial Version and Version 1 of Simtest

Timestamp:

Feb 24, 2009, 4:23:58 PM (17 years ago)

Author:

trac

Comment:

--

Simtest

@@ +1 @@
+==== Summary ====
+'''Simtest:''' is a process for testing the IPP with simulated data.
+The simulated data is produced and injected into an SQL database by ipp_simulation_data.pl
+It can then be carried through the following ipp stages:
+
+ * Registration
+ * Detrend construction
+ * Chip processing
+ * Camera processing
+ * Warp to skycell
+ * Difference
+ * Stack
+
+----
+Updated for ipp-2.4 / ipp-2.5 by S.Rodney
+----
+=== Annotated simtest walkthrough ===
+
+
+
+==== Configuration: ====
+ * Check ~/.ipprc (or $PS_SITE) 
+  * SITE and SYSTEM point at valid site.config and system.config files
+  * PATH should contain your config directory
+ * '''site.config''': DATAPATH for SIMTEST, SIMWORK
+  * the directory pointed to by SIMTEST is where your RAW simtest data will be created and stored. You can refer to it in command -line arguments as path://SIMTEST/
+  * the SIMWORK directory is where the intermediate and final output files and all log files will be placed. Refer to it as path://SIMWORK/.  
+   * (The variable name used, e.g. SIMTEST or SIMWORK, is unimportant. Just be consistent in the program calls below)
+  * DBSERVER, DBNAME, DBUSER, DBPASSWORD
+   * these set the parameters used by programs to access the SQL database
+
+ * Check ~/.ptolemyrc for:
+  * CONFDIR
+   * this is the directory containing your configuration files.
+
+==== Initiating a database. ====
+ *  There are three ways: 
+1) If you don't care to use the ippMonitor webpage, you can create an empty database in mysql
+ > mysql
+ mysql> CREATE DATABASE mydatabase;
+ mysql> exit 
+then initiate the new database
+ > pxadmin -create -dbname mydatabase
+
+2) If you are restarting an existing database project you can delete and rebuild an empty simtest database:
+ > pxadmin -recreate -dbname mydatabase
+
+3) You can use dbadmin to create an empty database that will be recognized by ippMonitor (note: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere. Call dbadmin with no options for syntax help.)
+ > cd /data/po02.0/ipp/ipp-2.1/ippMonitor
+ > dbadmin[[BR]]
+ USAGE: dbadmin (options)[[BR]]
+  dbadmin init (dbserver) (dbuser) (dbpassword)
+      creates admin tables, basic db user -- requires root password[[BR]]
+  dbadmin client (dbserver) (dbuser) (dbpassword) (client)
+      add client name or regex[[BR]]
+  dbadmin user (dbserver) (dbuser) (username) (password)
+      create a new www user and password[[BR]]
+  dbadmin project (dbserver) (dbuser) (dbname)
+      create a new ipp project (database)[[BR]]
+ > dbadmin project alala ipp mydatabase
+
+then initiate the new database
+ > pxadmin -create -dbname mydatabase
+
+==== Make sim data ====
+'''(and inject it into the database)'''
+ > ipp_simulation_data.pl --name test --dbname mydatabase --camera SIMTEST --telescope SimScope --workdir path://SIMWORK/ --path path://SIMTEST/
+
+wait until it finishes ...
+
+==== Start pantasks ====
+pantasks is the ipp parallel process manager for distributed computing across multiple nodes.
+start up pantasks from a second window
+ > pantasks
+ 
+ Welcome to pantasks - parallel task scheduler
+ 
+load some pantasks commands
+ pantasks: module pantasks.pro
+
+or, if you have a modified pantasks.pro file
+ pantasks: input /home/username/pantasks/pantasks.pro
+
+Add some CPU 'controllers' to carry out the processing steps.  Each controller you add will start an instance of the 'pclient' program, which receives processes from the pantasks 'pcontroller.' 
+To add an individual controller host by hand:
+ pantasks: controller host add myhost 
+
+'''NOTE:'''  You must have passphraseless ssh authentication set up from your pcontroller machine (the one running pantasks) to your pclient machines (the ones added with 'controller host add').  
+You may want to set up a module in your pantasks.pro file to quickly add a set of controllers with one command.
+
+If you are using a database other than the one declared in your site.config file, use the DB variable. DB holds a list of databases. $DB:0 is the active database. The default database defined in your site.config file will only be used if $DB:n is 0.
+ pantasks: list DB -add mydatabase
+ pantasks: echo $DB:0
+ mydatabase
+
+The pantasks.pro module should contain a function called '''add.database''' to do this as well.
+ pantasks: add.database mydatabase
+
+==== Registration ====
+The newly injected simulated data is in the database, but we don't know what each image is yet, so we register all the new exposures in the database, getting their vital statistics and sorting them appropriately. 
+In pantasks, start up the tasks list, turn off some switches and check the status
+
+ pantasks: module.tasks
+ pantasks: all.off
+ pantasks: register.on
+ pantasks: status
+ Task Status
+  AV Name                    Njobs  Ngood Nfail Ntime Command
+  -+ detrend.process.load        0      0     0     0 (dynamic)
+  -+ detrend.process.run         0      0     0     0 (dynamic)
+  -+ detrend.processexp.load     0      0     0     0 (dynamic)
+  -+ detrend.processexp.run      0      0     0     0 (dynamic)
+  -+ detrend.stack.load          0      0     0     0 (dynamic)
+  -+ detrend.stack.run           0      0     0     0 (dynamic)
+  -+ detrend.norm.load           0      0     0     0 (dynamic)
+  -+ detrend.norm.run            0      0     0     0 (dynamic)
+  -+ detrend.normexp.load        0      0     0     0 (dynamic)
+  -+ detrend.normexp.run         0      0     0     0 (dynamic)
+  -+ detrend.normstat.load       0      0     0     0 (dynamic)
+  -+ detrend.normstat.run        0      0     0     0 (dynamic)
+  -+ detrend.resid.load          0      0     0     0 (dynamic)
+  -+ detrend.resid.run           0      0     0     0 (dynamic)
+  -+ detrend.residexp.load       0      0     0     0 (dynamic)
+  -+ detrend.residexp.run        0      0     0     0 (dynamic)
+  -+ detrend.reject.load         0      0     0     0 (dynamic)
+  -+ detrend.reject.run          0      0     0     0 (dynamic)
+  -+ chip.imfile.load            0      0     0     0 (dynamic)
+  -+ chip.imfile.run             0      0     0     0 (dynamic)
+  -+ camera.exp.load             0      0     0     0 (dynamic)
+  -+ camera.exp.run              0      0     0     0 (dynamic)
+  -+ warp.exp.load               0      0     0     0 (dynamic)
+  -+ warp.exp.run                0      0     0     0 (dynamic)
+  -+ warp.skycell.load           0      0     0     0 (dynamic)
+  -+ warp.skycell.run            0      0     0     0 (dynamic)
+  -+ diff.skycell.load           0      0     0     0 (dynamic)
+  -+ diff.skycell.run            0      0     0     0 (dynamic)
+  -+ stack.skycell.load          0      0     0     0 (dynamic)
+  -+ stack.skycell.run           0      0     0     0 (dynamic)
+  ++ register.imfile.load        0      0     0     0 (dynamic)
+  ++ register.imfile.run         0      0     0     0 (dynamic)
+  ++ register.exp.load           0      0     0     0 (dynamic)
+  ++ register.exp.run            0      0     0     0 (dynamic)
+
+Start the registration step
+ pantasks: run
+
+You can check the progress of the registration with status calls or on the web monitor: http://kiawe.ifa.hawaii.edu/ippMonitor
+
+If you are watching the processes on your controller hosts (e.g. using 'top') then you should see pclient processes.  The pantasks process itself is called pcontrol.
+
+When all your images have finished registration you can halt the register.load processes, which are no longer needed. (or they can continue to run in the background.)  You can turn off the registration switches.
+ pantasks: stop
+ pantasks: register.off
+
+'''NOTE:''' using 'halt' instead of 'stop' will stop the processes without harvesting jobs.
+
+==== Detrend construction ====
+Flip switches for the detrend step
+ pantasks: detrend.on
+
+Back in a base shell, outside of pantasks, run the detrend tool. This creates a detRun table instance in the database, from which pantasks can get the info it needs to direct the detrend construction
+ > dettool -dbname mydatabase -definebyquery -det_type BIAS -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type BIAS
+
+You should now be able to find an entry for this detrend run on the ippMonitor web page or with a MySQL query on the detRun table. When you see it there, start the pantasks scheduler in your pantasks window, window 2.
+ pantasks: run
+
+When you see the bias in the ippMonitor under Detrend Frames or in MySQL under detFrames, then go back to window 1 for a sequence of dettools for DARKs, SHUTTERs, and FLATs. After running each dettool command from the command line, you should be able to find the run on the ippMonitor or in MySQL. 
+ > dettool -dbname mydatabase -definebyquery -det_type DARK -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter NONE -select_exp_type DARK
+
+Wait for the dark to finish before starting to make the shutter
+ > dettool -dbname mydatabase -definebyquery -det_type SHUTTER -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
+
+Wait for the shutter to finish before starting the flats (but both flats can be done at once)
+ > dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter r -select_exp_type FLAT -select_filter r
+
+ > dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope -filter i -select_exp_type FLAT -select_filter i
+
+
+When detrend creation is done, you can stop and switch off the detrend processing.
+
+ pantasks: stop
+ pantasks: detrend.off
+
+==== Chip and Camera ====
+Start the chip and camera processing. This takes all the science images and does the detrending (dark sub, bias sub, shutter correction, flat fielding)
+
+<pre>
+ pantasks: chip.on
+ pantasks: camera.on
+ pantasks: run
+
+ ...
+
+ pantasks: stop
+</pre>
+
+==== Warp ====
+Next we prepare for warping the science images into common skycells. i.e. putting them on a common tessellated tangent plane projection
+
+If the skycells have not been defined, then the skycells program will make them. This creates a directory with catalogs of skycell definitions.
+ > skycells -D CATDIR ./skycells 8 -scale 0.2
+  no data in db ./skycells/Photcodes.dat
+  directory ./skycells does not exist, creating...
+
+'''IMPORTANT:''' be sure to point to this directory in your site.config file with, e.g.:
+ TESSELLATIONS   METADATA
+        # Tessellation name with corresponding DVO catdir
+        Skycells        STR     path://SIMTEST/skycells
+ END
+ 
+Now we should be able to go back to the pantasks window and start warping:
+
+<pre>
+ pantasks: warp.on
+ pantasks: run
+
+ ...
+
+ pantasks: stop
+</pre>
+
+
+The warped images will be stored in path://SIMWORK/tess_skycells/sky_skycell.<skycellnum>/ as '*.wrp[1-7].fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats').  The warping will automatically calculate the appropriate "<skycellnum>" for the given tessellation and location of image on the sky.
+
+==== Difference ====
+
+<pre>
+ pantasks: warp.off
+ pantasks: diff.on
+ pantasks: run
+
+ ...
+
+ pantasks: stop
+</pre>
+
+These difference image will be stored in path://SIMWORK/tess_skycells/sky_skycell.0743078/ as '*.dif1.fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats').
+
+==== Stack ====
+
+<pre>
+ pantasks: diff.off
+ pantasks: stack.on
+ pantasks: run
+
+ ...
+
+ pantasks: stop
+</pre>
+
+Lather, rinse, repeat.