summit copy issues / pzgetimfiles should fault on error

[eugene]

short story: pzgetimfiles does not always set a fault in the db when it fails.

longer story:

I've discovered the proximate cause for the recent crashes of the
summit copy pantasks. It seems that there is a single summit exposure
(o5602g0593d) for which the file index is buggered up. Instead of a
clean listing of files, we have a large number of camera communication
messages at the top of the index:

# fileID |bytes |md5sum |type|chipname|

Observing Log

Observers

Tommy and Bill

Starting Conditions

==Starting Conditions==
*Ugly: humidity 82%, wind speed 9.6 m/s, ground fog blowing up out of the crater
*Otisgui restarted; telescope unpinned; Estop cleared
*Tried to start Az. drive ex. script at 5:45 anyway but got error
message: "Failed to get Telescope status"
*Software updated; exercise script started at 6:08
*Conditions clearing; restarted az exercise script with 15 sec pause
between moves
*6:45 opened vents; az drive 30 deg

Tonight's Observing Plan

(et cetera)

Further down in the file, there is an actual list corresponding to a
fileset. But it, too, is buggered up:

c4741g0016d56.fits|49423680|7e22d2cf9d1c722721a7868ca88c41da|chip|ota56 |
o5410g0021o35.fits|49432320|48a67d5d416b6dcfdb1552c3417f583f|chip|ota35 |
o5602g0593d01.fits|49429440|e4ca8b9284bba1d83b79b28a3a2cb765|chip|ota01 |
o5602g0593d02.fits|49429440|dd1ce4d2780d8ab50bdf5c0f0dc88f0d|chip|ota02 |
o5602g0593d03.fits|49429440|ea85652366d88fe5c1a1a900164596e8|chip|ota03 |
o5602g0593d04.fits|49432320|f5ad936554987638b6e00a6c7df0bd27|chip|ota04 |

In addition to the correct (and complete) set of files, the listing
includes 2 files from earlier exposures (much earlier -- like 6 months
and 2.4 years old!).

When pzgetimfiles is run on this entry, it spews back a bunch of
complaints to stderr about the lines that it does not recognize. It
then attempts to parse the lines it does recognize but, as a result of
the extra files, it raises an error and exits. But, in the particular
error case, it does NOT set a fault on the summit exposure. Thus, it
keeps trying to download this same index every so often -- 9242 times
since this problem file appeared on Feb 10. We know that
pantasks_server or pcontrol will eventually get unhappy and crash if
there is too much stderr and/or stdout traffic which it must parse, so
this eventually kills it (which ever one it is that is crashing).

So, a summary of the bugs that are revealed by this situation:

1) broken summit datastore index file :
​http://conductor.ifa.hawaii.edu/ds/gpc1/o5602g0593d/index.txt
2) pzgetimfiles does not raise an error on any / many parsing errors
in an index [note 1]
3) pzgetimfiles does not validate the filename against the exposure
name [note 2]
4) pzgetimfiles does not always fault the summitExp table on failures
5) pantasks_server and/or pcontrol should not crash under excess
stdout / stderr communications

note 1 : it is not clear that pzgetimfiles *should* raise an error if
it fails on any lines. that level of rigidity could be good (would
detect problems) or it could be bad (would cause things to fail in
potentially recoverable cases).
note 2 : it is not clear that pzgetimfiles should attempt this file
name validation : this forces pzgetimfiles to have knowledge of the
relationship between file names which would restrict its range of
usability

I have manually faulted the bad exposure. it is a dark, so it is
acceptable that the download fail forever.

aloha
gene