Changes between Initial Version and Version 1 of Pantasks_FAQ

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

Pantasks_FAQ

@@ +1 @@
+==== What is pantasks? ====
+pantasks is the ipp parallel process manager for distributed computing across multiple nodes. Also see the [wiki:IppTools_FAQ ippTools FAQ] for information on some of the commands that get launched by pantasks.
+
+==== How do I start pantasks? ====
+start up pantasks from a terminal window
+ > pantasks
+ 
+ Welcome to pantasks - parallel task scheduler
+ 
+load some pantasks commands
+ pantasks: module pantasks.pro
+
+or, if you have a modified pantasks.pro file
+ pantasks: input /home/username/pantasks/pantasks.pro
+
+After loading the pantasks.pro file, you can add a database easily:
+ pantasks: add.database mydatabase
+
+If you don't add a database, pantasks will use the one declared in your .ipprc file.
+
+==== How do I configure pantasks? ====
+
+.pantasksrc
+.ptolemyrc
+
+==== What are the primary pantasks commands? ====
+
+Connect to a controller host
+ pantasks: controller host add myhost
+
+Check the controller host status (NOTE: This will sometimes return no info, even when there are active hosts):
+ pantasks: controller status
+
+Check the processing status:
+ pantasks: status
+
+For additional timing details:
+ pantasks: status -taskstats
+ 
+Start processing:
+ pantasks: run
+
+Stop adding new processes, but finish out the queue :
+ pantasks: stop
+
+Stop processes right now :
+ pantasks: halt
+
+Exit pantasks :
+ pantasks: exit
+
+
+==== How do I get more verbose output from pantasks? ====
+
+ pantasks: $VERBOSE = 1
+
+Raise the number above 1 for more and more verbosity.
+
+==== Why does my process fail in pantasks but succeed on the command line? ====
+
+"I copied the command directly from the pantasks error stream (or from the verbose command output) and pasted it into a separate terminal.  It succeeds on the terminal, but it failed in pantasks." 
+
+You may have a config error in your home directory.  When pantasks executes a command, it does so from the user's home directory on whichever remote host happens to have been assigned for that process.   If you happen to have some out of date config directories in that home directory, then they may be loaded '''before''' the system-level config directories.  Here is an example: 
+In your .ipprc file, you may have defined the path to your configuration directory with something like this:
+
+ PATH            STR     /path/to/my/system/level/ippconfig
+
+So in your system.config file you can define the directory for your GPC1 camera.config file with this line: 
+
+ CAMERAS                METADATA
+        GPC1                    STR     gpc1/camera.config
+ END
+ 
+so when you run any script that needs to reference the GPC1 camera it will look 
+ * first in the current directory for ./gpc1/camera.config
+ * next in the directory defined by your .ipprc PATH variable (in this case /path/to/my/system/level/ippconfig/gpc1/camera.config)
+
+Thus, if you have some old gpc1/camera.config lying around in your home dir, then pantasks will look there first, and will hit a config error that would not appear if you run the command from the command line when you are not in your home dir.
+
+'''Solution:''' move any old config files out of your home directory, or update them to remove the config issue.
+
+==== Why are my nodes "down" or "resp"? ====
+
+ * First, check that you can ssh from the machine on which you are running pantasks to the node without being prompted for a password and without errors reading your shell startup file (.bashrc, .cshrc, .profile, .login and the like):
+  * Try "ssh myhost"
+  * If you're prompted for a password, then you need to set up ssh keys, and/or check your ssh configuration.
+ * Second, check that you can start up 'pclient' over an ssh connection. For that to work, you need to run 'psconfig ipp-2.6.1' (or whatever the IPP version is called on your system) in your startup file. 
+  * Try "ssh myhost pclient". If it works, you can exit pclient with "exit" or "quit".
+  * If it doesn't work, you need to check your shell configuration (.cshrc or .bashrc) to ensure the IPP environment is being loaded via psconfig. In .cshrc, just make sure that 'psconfig ipp-2.6.1' is executed. In .bashrc, there's a catch: bash doesn't let you use an alias in the same file that defines it, so you need to expand the 'source'  by hand in .bashrc, thus (this fix will be included in the INSTALL instructions in releases post 2.6.1.): 
+ if [ -f /IPP/psconfig.csh ]; then
+    alias psconfig='source /IPP/psconfig.bash'
+    source /IPP/psconfig.bash ipp-2.6.1
+ else
+    alias psconfig='echo psconfig not available'
+ fi
+ * If there is a long delay between executing the "ssh" command and the shell appearing, there may be a timeout problem.
+  * Make sure your shell configuration is not too complex.
+  * One bash user had the IPP environment being set up in both .bashrc and .cshrc, so that psconfig was being called first when bash started, and then again on starting psconfig (which is a csh script).  This produced a long delay, causing the command to time out in pantasks.
+ * Finally, this can also be triggered by the readline bug, whose fix is described below under [wiki:#detrend_resid_imfile.pl_triggers_the_error_message_'Unknown_option:_--erbose' detrend_resid_imfile.pl triggers the error message 'Unknown option: --erbose']
+
+==== On startup I get "can't find config file. some functions will be unavailable."  Which config file is missing? ====
+
+You're missing the .ptolemyrc file.   Copy dvo.site from your site-level config directory into your home dir and rename it as .ptolemyrc.
+
+==== How do I re-run files that have failed at some stage after fixing the bug that caused the failure? ====
+
+Most of the <code>ippTools</code> binaries (which provide the database interaction) have some version of a <code>-revert</code> command.  For example, if a camera stage failed, try <code>camtool -revertprocessedexp -cam_id 12345</code>.  You can also revert based on an error code using the <code>-code</code> argument.
+
+==== Process X succeeded without fault, but it is not moving on to process Y.  How do I force it to proceed? ====
+
+First verify that none of the sub-stages failed.  For example, if a chipRun has state 'new' and you think it should be 'full' and moving on to camRun, then check to be sure that all of the contributing chipProcessedImfile rows in your database are completed with fault=0.   Then check to be sure that the next process in line was not initiated.  e.g. Do you have a camRun with the chip_id that you expect? 
+
+If the process really just stopped without raising a fault and without initiating the next stage, then you can try manually setting its state to 'full' and using the appropriate ippTool to initiate the next process in the sequence.   For a stalled chipRun with chip_id 2591, this would be done like this:
+
+<pre>
+chiptool -dbname myDatabase -updaterun  -label 'myLabel' -chip_id 2591 -state full    
+
+camtool -dbname myDatabase -definebyquery -chip_id 2591 -set_label 'mylabel' 
+</pre>
+
+
+==== detrend_resid_imfile.pl triggers the error message 'Unknown option: --erbose' ====
+
+The dropped character in a long line is a classic symptom of a bad 'readline' library.
+To fix it, do:
+<pre>
+% pschecklibs -build -force libreadline
+</pre>
+There should be no need to rebuild the IPP (since we're using dynamic libraries)
+
+
+==== I've got a job sitting in the queue, and a bunch of idle hosts... how do I get the job onto a host? ====