IPP Progress Report for the week 2011.11.14 - 2011.11.18

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• IPP: Installed and tested new tag with condor (validated by Heather)
• IPP: Czar on Thursday
• IPP: Helped Chris with ppMops
• IPP: Fixed deleted entries in magicDSFile with Bill
• PSPS: getting access to the production cluster, making it work, attempting to monitor...
• IPP: Condor for MD06

Heather Flewelling

• sick thursay
• czar for many days - lots of bad weather
• addstars
  • wrote up wiki on addstar upgrades ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/AddStarUpgradeNov2011
  • czar for many days - lots of bad weather
  • restarted dvodb merging - ThreePi is merging onto a copy of Gene's dvodb on ipp045
  • stopped dvodb merging to investigate aftermath of db computer reboot
  • restarted LAP merging, ThreePi still undergoing tests
  • added safety features to merging to stop merging after a faulted minidvodb merge
• liason
  • downloaded md fields for tom
  • started working with eric hilton on skyprobe
• gave serge some chips for condor testing
• investigated staticsky problem which bill fixed

Roy Henderson

• sick Mon/Tues/Wed

• changes to improve efficiency of ODM polling code: now checks unloaded, then non-mergeworthy, then unmerged in order to avoid checking same batch twice
• investigated weird case of duplicate batches on the datastore. Lots of help from Bill.
• painstakingly checked md5sums of other batches that reported a failure on datastore. All ok.
• moved OLD processing to Heather's new DVO database on ipp005
• lots of meetings and discussion regarding code changes for ORR
• started to code changes required for stacks

Mark Huber

• psphotstack test runs for SAS, footprint, MD04 sample
• MD06 reference and test stack prep
• little time spent continuing to trace through ppStack deep stack convolution problem

Bill Sweeney

• Updated the destreak stage scripts and magicdstool to disambiguate between runs that fail to revert during update and inital processing.
• Spent many hours recovering database rows lost because of an ippTools bug that I introduced recently.
• Investigated a problem with the data store reported by Roy. The registration code ran multiple times during a period where ippc17 was crashing and multiple versions of a fileset got registered with the same name. This bother psps. The problem is due to a missing constraint in the tables. (When I wrote these many years ago my sql was rusty.)
• Fixed a problem with staticsky distribution. Multiple runs were getting queued due to some bugs in the SQL.
• Corrected a number of exposures that had inconsistent data_states that prevented updates from proceeding.
• Set up scripts for queuing M31 data to be processed and distributed. Started processing all data from 2010 without using magic.
• Fixed a bug in raw stage distribution. Started distributing all M31 raw data
• With Gene's help, debugged and fixed an assertion failure in ppImage.

Chris Waters

• SSTF: Ran tests with different detection limit and with convolution turned off in the diff. Worked on tracking down bugs in unconvolved diffs that have not been identified previously. Both of these increase the number of detections over the default processing, and enabling both results in a large increase.
• LAP: More documentation (​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/LAP_Czar). Reduced number of simultaneous runs.
• Diskspace: Identified miscounting problem in diskspace accounting.
• Cluster hardware: Added new wave 4 nodes into nebulous and pantasks processing lists. Added new ATRC node into nebulous and rsynced data from ippb02 to ippb03 to keep ippb02 from being completely full (which has unintended consequences for shuffle and processing due to the high low it creates).
• Sky background: further tests to reduce the number of cells that need to be modified by the pattern correction (which seems to result in better background results).