Changes between Initial Version and Version 1 of diff_fixits

Timestamp:

Sep 7, 2010, 9:55:50 AM (16 years ago)

Author:

bills

Comment:

--

diff_fixits

@@ +1 @@
+
+{{{
+We have 2 diff faults today. Let's fix them.
+
+***************************************
+
+diff_id 76390 skycell.1194.140 fails because it can't read input warp file on ipp045 
+(This was a node that had some troublesome days last month)
+
+Confirm that file is corrupt with
+
+    funpack -S /data/ipp045.0/nebulous/2d/e8/391435564.gpc1:SweetSpot.nt:2010:08:08:o5416g0096o.204249:SR_o5416g0096o.204249.wrp.90720.skycell.1194.140.wt.fits > broken.fits
+
+    FITSIO status = 108: error reading from FITS file
+    Error reading data buffer from file:
+    /data/ipp045.0/nebulous/2d/e8/391435564.gpc1:SweetSpot.nt:2010:08:08:o5416g0096o
+    .204249:SR_o5416g0096o.204249.wrp.90720.skycell.1194.140.wt.fits
+    Error reading elements 1 thru 2147418112 from column 1 (ffgclb).
+    error reading compressed byte stream from binary table
+
+Yep it's toast.
+
+To regenerate the file execute the new script that I am going to finish and add to the tools directory.
+(Or perhaps I will build the feature into warp_skycell.pl)
+
+    perl runwarpskycell.pl --warp_id 90720 --skycell_id skycell.1194.140 --no-update
+
+no-update because we don't want to update the database (The warpRun is full. The warpSkyfile is in the database.
+We just want to fix the bits)
+
+Unfortunately runwarpskycell fails because input chips not found.
+
+This is sweetspot data from early August being diffed against the stacks.
+
+To find out why the data can't be found use
+
+    chiptool -processedimfile -chip_id 121251 -class_id xy52
+
+to find the state of one of the required inputs.
+
+We find chipRun has been cleaned (The data was previously processed as warp warp diffs. We've been
+keeping the warps around to run against the stacks which were made at the end of the month)
+
+We can get the chip images back with
+
+ chiptool -setimfiletoupdate -chip_id 121251 -set_label update
+
+For extra credit we could use warptool -scmap -warp_id 90720 -skycell_id skycell.1194.140 to find
+the subset of chips to process, but I didn't. We'd queue them for updates 1 at a time by
+adding -class_id xy?? to 
+     chiptool -setimfiletoupdate -chip_id 121251 -set_label update -class_id xy52
+     chiptool -setimfiletoupdate -chip_id 121251 -set_label update -class_id xy53
+     etc.
+
+Since we'll clean the data soon why bother.
+
+Since magicked == -1 destreaking is necessary for the warps to run so
+
+       magicdstool -updaterun -set_state new -set_label update -magic_ds_id 215998
+
+Wait for updates to complete. 5 minutes or so.
+
+Then run the runwarpupdate.pl script. Success
+
+Now revert the diff failure
+    difftool -revertdiffskyfile -diff_id 76390
+
+A few minutes later the diff is complete
+
+*****************************************
+Case 2.
+
+?????????????????????????????????????????????Assertion failed in function psThreadLauncher at psThread.c:244. Error stack:
+Unable to perform ppSub: 4 at /home/panstarrs/ipp/psconfig//ipp-20100823.lin64/bin/diff_skycell.pl line 400.
+Running [/home/panstarrs/ipp/psconfig/ipp-20100823.lin64/bin/difftool -diff_id 76373 -skycell_id skycell.1460.009 -fault 4 -adddiffskyfile -dtime_script 62.9999801516533 -hostname ipp053 -path_base neb://ipp053.0/gpc1/SweetSpot.nightlyscience/2010/09/07/RINGS.V0/skycell.1460.009/RINGS.V0.skycell.1460.009.dif.76373 -dbname gpc1]...
+
+This is a recurring bug at the diff stage. Ticket #1422 covers it.
+
+I have had success fixing these by running the command by hand without threads.
+
+They also sometimes succeed after reverting. Let's try that
+
+    difftool -revertdiffskyfile -diff_id 76373
+
+Wait few minutes
+
+ difftool -diffskyfile -diff_id 76373 -skycell_id skycell.1460.009 | grep state
+ data_state       STR       full            
+ state            STR       full
+
+It worked. It would be nice to fix this bug.
+}}}