Context Navigation

Changes between Version 41 and Version 42 of Processing

Timestamp:: Apr 13, 2010, 4:30:12 PM (16 years ago)
Author:: rhenders
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

Processing

-              v41
+              v42
 = Finding and dealing with errors =
 As shown above, the {{{pantasks: status}}} command shows failures for a particular processing stage. A handy script also exists for monitoring all stages for a particular label (basically a shortcut for using the 'Night summary' page of the ippmonitor (see above)). It is found in {{{/tools}}} under the build directory. Example usage is shown below.
+As shown above, the {{{pantasks: status}}} command will display failures for a particular processing stage. A handy script also exists for monitoring all stages for a particular label (basically a shortcut for using the 'Night summary' page of the ippmonitor (see above)). It is found in {{{/tools}}} under the build directory. Example usage is shown below.
 {{{
 …
 }}}
 The output highlights a problem at the diff stage, and in the {{{faults and log files}}} section, the relevant log file is listed, which can be views using {{{neb-tail}}} as below.
+The output highlights a problem at the diff stage, and in the {{{faults and log files}}} section, the relevant log file is listed, which can be viewed using {{{neb-tail}}} as below.
 {{{
 …
 }}}
+== Common issues ==
+This section attempts to outline common issues encountered during processing and how to work through them.
+=== stdscience ===
+'''Chip failures''', for example
+{{{
+  AV Name                     Nrun   Njobs   Ngood Nfail Ntime Command
+  ++ chip.imfile.run             0   23536   17755  5781     0 chip_imfile.pl
+}}}
+To investigate the failures, go to
+[http://ipp004.ifa.hawaii.edu/ippMonitor/Login.php ippmonitor]->Science steps->Chip Failed Imfiles
+where you can view the logs by clicking within the 'State' column.
+'''Warp failures'''
+To investigate the failures, go to
+[http://ipp004.ifa.hawaii.edu/ippMonitor/Login.php ippmonitor]->Science steps->Warp Failed Skyfiles
+Filter results by using 'new' in the state column. For the results, check that the values in the 'Fault' column are 2, which denotes an NFS error, in which case we can 'revert' using
+{{{
+pantasks: warp.revert.on
+}}}
+Remember to switch off again afterwards with
+{{{
+pantasks: warp.revert.off
+}}}
+= Rebuilding the IPP code =
+The IPP in use presently is located at
+{{{
+~ipp/ipp-20100211
+}}}
+If the code needs an update and rebuild, then:
+ * stop pantasks (as above)
+ * {{{cd ~ipp/ipp-20100211}}}
+ * {{{svn update}}}
+ * {{{psbuild -dev -optimize}}}
+ * restart pantasks (as above)
+= Who to contact =
+Any problems or concerns should be reported to the ipp development mailing list:
+{{{
+ps-ipp-dev@ifa.hawaii.edu
+}}}
+Different members of the IPP team are responsible for different parts of the code, and the relevant person will hopefully address the issue.
+= Fault states =
+Another script in the {{{/tools}}} directory can be used to probe errors more thoroughly. Chip, camera, warp, stack, diff, magic and destreak are currently supported, as well as the use of multiple labels (all use LIKE so wildcards work). You can also search by a specific fault code, or limit the query. The script categorises errors for entries at the specified stage and with the provided label. The output is the appropriate id and component, the machine it was run on, and the particular problem (e.g., a file that failed to be found, otherwise the resolved name of the processing log), all grouped into categories.  With such a list, it's easy to identify patterns, e.g., a few warps are failing because of a single corrupt camera mask file; or machine X can't read files on machine Y. For example, using the same label as above:
+{{{
+./errors.pl --dbhost ippdb01 --dbuser ipp --dbpass ipp --dbname gpc1 --label 'SweetSpot.20100409' --stage diff
+}}}
+Will produce
+{{{
+Total: 1
+Assertion failures: 1
+.skycell.1500.091(ippc08): /data/ipp034.0/nebulous/61/ef/244153882.gpc1:SweetSpot.20100409:2010:04:12:RINGS.V0:skycell.1500.091:RINGS.V0.skycell.1500.091.dif.50316.log
+}}}
+The 'fault codes' mentioned above are as follows.
+|| '''Code''' || '''Description''' ||
 || 1 || Error of unknown nature ||
 || 2 || Error with a system call (often an NFS error) ||
 …
 || 6 || Error due to timeout ||
+== Common issues ==
+This section attempts to outline common issues encountered during processing and how to work through them.
+=== stdscience ===
+'''Chip failures''', for example
+{{{
+  AV Name                     Nrun   Njobs   Ngood Nfail Ntime Command
+  ++ chip.imfile.run             0   23536   17755  5781     0 chip_imfile.pl
+}}}
+To investigate the failures, go to
+[http://ipp004.ifa.hawaii.edu/ippMonitor/Login.php ippmonitor]->Science steps->Chip Failed Imfiles
+where you can view the logs by clicking within the 'State' column.
+'''Warp failures'''
+To investigate the failures, go to
+[http://ipp004.ifa.hawaii.edu/ippMonitor/Login.php ippmonitor]->Science steps->Warp Failed Skyfiles
+Filter results by using 'new' in the state column. For the results, check that the values in the 'Fault' column are 2, which denotes an NFS error, in which case we can 'revert' using
+{{{
+pantasks: warp.revert.on
+}}}
+Remember to switch off again afterwards with
+{{{
+pantasks: warp.revert.off
+}}}
+= Rebuilding the IPP code =
+The IPP in use presently is located at
+{{{
+~ipp/ipp-20100211
+}}}
+If the code needs an update and rebuild, then:
+ * stop pantasks (as above)
+ * {{{cd ~ipp/ipp-20100211}}}
+ * {{{svn update}}}
+ * {{{psbuild -dev -optimize}}}
+ * restart pantasks (as above)
+= Who to contact =
+Any problems or concerns should be reported to the ipp development mailing list:
+{{{
+ps-ipp-dev@ifa.hawaii.edu
+}}}
+Different members of the IPP team are responsible for different parts of the code, and the relevant person will hopefully address the issue.
 {{{magic}}} has fault states greater than 5.