IPP Progress Report for the week 2011.07.11 - 2011.07.15

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Nebulous mysql dump now works fine again
• Changed some mysql dump configuration (isp and ippadmin dumps were still done from ippdb01)
• SVN repository for operational configuration + associated control
• IPP Czar on Thursday and Friday
• IPP Test Framework meeting with Jim, Gene, Mark, and Roy.
• Early IPP Test Framework developments

Heather Flewelling

Roy Henderson

• ippToPsps loading:
  • mangled skycell IDs (see last week's report) broke the load. Fudged numbers, then reloaded stacks
  • loaded a few more detections and stacks from DVO
  • merge completed and I checked that I could query the new data. Everything looked OK
  • some help for Kent Wood prior to old 3PI deletion. I think he and his team succeeded in extracting what they need.

• ippToPsps development:
  • more work on addRun SQL query. Was getting un-merged DVO items. Now, with Heather's help, it gets only merged/full items
  • main program now polls indefinitely, looking for new P2 or ST items to queue up. This has been running all week with no problems

• Czar Tuesday and Wednesday:
  • lots of errors, some failed attempts to revert things manually
  • ipp026 went crazy and managed to halt the whole pipeline. Twice.

• Czartool:
  • cleaned-up webpage with lots of changes to tables and colors for clarity
  • page now has two 'modes', standard and update, so we don't need two big tables cluttering up the page
  • improved query to get magic mask fraction with help from Chris. Now much faster.
  • changes to code for time-series plots, including...
  • a new rate plot suggested by Gene that uses a running-mean to show exposures-per-hour more clearly than before

• started work on populating a gpc1 database instance for the test loop stuff

Mark Huber

• Implementing MD10.V3 tessellation. Stumbled across bug in pswarp Gene was able to fix. Setup and in use now for MD10 observations. Exposures in y-band run through warp stage for MD10.V3 reference stack.
• simtest->PSPS development - basic sample for loading into PSPS generated.
• Continued desktop setup/configurations.

Bill Sweeney

Chris Waters

• Dark: Worked on ways to select which order of model is supported by the data in an attempt to only fit quadratic models in temperature when that would not introduce more noise into the dark. This still seems to introduce more noise than a simple linear in temperature model (which has similar introduced noise characteristics as the old linear dark model). Plotting up pixel flux values from a wide range of exposure times and temperatures shows no obvious quadratic term, even in pixels where the quadratic term has been selected by the code. This is also true in corner glow areas, which appear to be linear in temperature as well. Currently constructing a full exposure model where the quadratic term is excluded, in an attempt to see if this is required at all for any OTA.
• Nebulous: Confirmed that in some cases, the shuffle code is failing to move an instance of the raw data to the long term storage nodes at the ATRC. However, this is not resulting in data loss, as this failed instance (which has zero size) is being detected and the valid second copy is not deleted. Updated neb-stat to identify these files by calculating md5sums for each instance that is available. Corrected shuffle SQL to properly select all files that should be moved.
• LAP: preliminary work to allow quickstacks to be skipped if only a small fraction of the input exposures are unpaired. This should speed up LAP processing at the expense of a small number of input exposures (usually ~3 / 50-60 total inputs).