PS1 IPP Czar Logs for the week 2011.01.10 - 2011.01.16

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.10

Bill is the acting czar for the next two days

• 08:30 All of last night's science data is through warp. Ran nightly_science.pl --queue_stacks --date 2011-01-10 --dbname gpc1
• 14:40 added label DS.RECOVER.TEST to stdscience. 1 exposure to be processed to test destreaking with saving recovery pixels.

Tuesday : 2011.01.11

• 09:00 sts dist run 377773 was stuck because of a corrupt file: the variance image from warp_id 144635 skycell.379. Due to the complexity of the destreaking process it isn't easy to regenerate this file. To allow the distRun to proceed skipping the broken skycell I performed the following steps.

1. in distribution pantasks: dist.off
2. disttool -revertcomponent -dist_id 37773
3. In mysql
       update warpSkyfile set quality = 42 where warp_id = 144635 and skycell_id = 'skycell.379';
4. In distribution pantasks: dist.on
5. wait a few minutes for distRun to complete
6. In mysql
       update warpSkyfile set quality = 0 where warp_id = 144635 and skycell_id = 'skycell.379';

Since quality of the skycell is non-zero distribution doesn't require the image to be readable. 
This allows the bundles for the other 87 skycells to get out the door. 

Since we put the quality back to zero if the warpRun is cleaned up the data will be available again through the update process.

• We have two nightly stacks that are stuck due to 2 skycells that fail with programming errors.

stack_id 214086 fails with the known problem

    Data error code: 32cf
-> pmPSFEnvelope (pmPSFEnvelope.c:327): unknown psLib error
     No fake sources are suitable for PSF fitting.
 -> ppStackPSF (ppStackPSF.c:54): Problem determining PSF
     Unable to determine output PSF.
 -> ppStackPrepare (ppStackPrepare.c:274): Problem determining PSF
     Unable to determine output PSF.
 -> ppStackLoop (ppStackLoop.c:72): Problem determining PSF
     Unable to prepare for stacking.

stack_id 215251 fails with
    Filtered out 81 of 964 sources
    fix this code: z1 should not be nan for 0.016674
    Backtrace depth: 7
    Backtrace 0: p_psAssert
    Backtrace 1: pmModelRadius_PS1_V1
    Backtrace 2: (unknown)
    Backtrace 3: (unknown)
    Backtrace 4: psThreadLauncher
    Backtrace 5: (unknown)
    Backtrace 6: clone
Unable to perform ppStack: 4 at /home/panstarrs/ipp/psconfig//ipp-20101215.lin64/bin/stack_skycell.pl line 354

• Except for these the two stacks the czartool dashboard is clear. But we are out of space.
• 10:55 stopped distribution pantasks while I debug destreak "goto_restored" mode. (Undo destreaking by swapping back in the backup files).
• With a change to magicdstool I was able to get "goto_restored" processing working but it doesn't reset the "magicked" bits back to zero. I took care of this by hand and queued all of the STS.201008 warpRuns that have been destreaked to be restored. It's a rather slow process.
• 12:30 queued new STS.201008 diffRuns with the correct reduction and label to STS.20101202.wait. Then changed the label to STS.20101202 where the inputs are from warpRuns that don't need to be restored. I'll change the label for the others once that process is complete. 179 of 573 are restored so far.

Wednesday : 2011.01.12

Thursday : 2011.01.13

Friday : 2011.01.14

Saturday : 2011.01.15

Sunday : 2011.01.16