IPP Progress Report for the week 2011.03.14 - 2011.03.18

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• MOPS czar
• mops helper for czar reporting
• IPP czar
• Nebulous replication on ippdb02
• pstamp backup
• Updated ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Processing with screen/czarpoll
• gpc1 replication failed

Heather Flewelling

• ippconfig test
  • ran magictest.3Pi.20110315
  • it is picking up the correct configs
  • comparing to magictest.3Pi.200110309.a - they have different recipes which use different masks.

• addstar
  • found a deadbeef in addtool -addminidvodbrun (I need to fix this still)
  • requeued up all the old minidvodbRuns for the new ThreePi database
  • started merging of old minidvodbs into new ThreePi.V2 database
  • added dvoverify checks and a few new columns for minidvodbProcessed. Commited and checked into tag and trunk.

• modified/copied diffgrep script in ipptrunk/tools to grep stack logs
  • sent gene a list of problematic(?) skycells in refstacks

• copied smfs to niall
• keep trying to process MD09.jtrp and MD10.jtrp - delayed by various problems

• sick 1.5 days

Roy Henderson

• PSPS
  • extracted more files for Jim's MD04 stack analysis (weights)
  • some discussions and planning with Jim regarding new plan to only publish detections seen twice or more.
  • PSVO
    • more design discussions with Daniel and Jim regarding integration of graphical query building into PSVO. We have a plan.
    • started on Classes to encapsulate database schema for the above
    • bug with JDBC drivers not on CLASSPATH for jar version: fixed

• IPP
  • czartool
    • a face-lift for czartool: nicer tables, clearer highlighting of errors, better general formatting
    • a fix for unsightly jumps in czartool plots when a new label is added (done by checking for large jumps in 1st derivative value, then adjusting accordingly)
    • finally added publish to czartool
    • created a new rate line plot (while retaining stacked histogram). Now using this on czartool webpage
    • added a 'pending postage stamp requests' table to webpage
    • added nebulous time series plot to webpage
  • ippToPsps
    • DetectionBatch class now utilizing new Fits class. Used new unit-test to ensure I didn't break anything along the way
    • added some documentation on detection batch unit-testing

• Other
  • some time granting data access to new Hungarian group

Bill Sweeney

• Spent a couple of days using psastro and DVO to perform and analyze tests with STS exposures that get poor astrometric solutions on some chips when using the standard settings. The conclusion was that Gene needs to do some work on psastro to solve the problem.
• Prototyped some code to track "Events" that occur during the lifetime of IPP output components. For example a given chip will be processed, cleaned up, updated to make postage stamps, cleaned up, updated to make postage stamps.... Often postage stamp jobs get stuck waiting for data that is in a long queue of things to be cleaned up to be finish just so that we can regenerate the image (or an image from another part of the exposure). We have found the need to track this in order to better manage our disk usage. Also if given images are in great demand we can stop the madness and keep those images around.
• We had performance problems with both of our databases this week. This caused lots of lost time investigating what was happening. In the end we restarted the database and things were fine for awhile.
• Spent two very busy days as processing czar.
• Wrote scripts to queue chip and diff processing for several hundred ThreePi exposures for which processing was incomplete. The chosen exposures are in the Hyades and nearby regions and these are of interest to Bertrand Goldman of MPIA and KP5.
• Created DVO database for all of the STS.2010 exposures to help with the astrometry investigations.
• Spent much of Saturday babysitting the processing listed in the last 2 items.

Chris Waters

• Investigated the dtime_photom values from the chip stage: ChipStage_Timing. Conclusion was that chips with bad dark models were identifying large numbers of faint sources, and the footprint culling algorithm ran slowly for these sources.
• Diskspace: Identified that hosts with larger-than-expected disk usage were hosts that had been offline at somepoint in the past. This causes them to have gaps in the range of so_ids that are available on that host, which decreases the effectiveness of the disk balance code. To get around this, disk balance now starts at a random so_id value to allow the full space to be explored with more regularity. Previously, only a small fraction starting at so_id = 1 were being examined.
• Reprocessing: organized scripts to calculate tessellation cells that are well populated in all filters, and plot DVO image footprints. This will help in choosing which data to use when we start reprocessing.
• Detectability server: bugfixes and rewrites to get this working correctly. Still not finished.