IPP Progress Report for the week 2012.02.06 - 2012.02.10

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Still populating ipp003 with nebulous dump (20120206).
• Ran slocate on the hosts. All index files are in <hostname>:/tmp/slocate. Note: slocate is different from one wave to another.
• mysql partitioning sucks (e.g. user-defined functions cannot be used for hash partitioning)
• Bad files hunt in nebulous and/or on ippb02 (with Heather):
  • Ran slocate on ippb02.[0-2]/nebulous (got ~10.7M files) -> This version runs incredibly fast. Scanning all three ippb02 disks took 22 minutes (no attempt to optimize)
  • Checked their status in nebulous:
    • ok
    • uri not present in instance table for a given ins_id: ~5M files (Note: it seems that some files are temporary) For a lot of those files, the uri in nebulous is ​file:///data/ippb02.1/nebulous[...] while the file on disk is ​file:///data/ippb02.0/nebulous[...].
    • uri not present in instance table and no ins_id but "similar" uris exist: 189k files

      Example of two similar uris:
      file:///data/ippb02.0/nebulous/00/00/1006632459.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      file:///data/ipp025.0/nebulous/00/00/1230393764.gpc1:ThreePi.nt:2011:01:23:o5584g0495o.285189:o5584g0495o.285189.ch.182968.XY34.mdl.fits
      

    • No entry in nebulous but there is a file on the disk: ~349k files (e.g: gpc1/ThreePi.nt/2010/12/30/o5560g0658o.269979/o5560g0658o.269979.ch.174390.XY03.ptn)
    Total: 5.7M "problems" See /tmp/nebulous_contents/problems.

Heather Flewelling

• ssp
  • ingested jt's ssp data into ssp database. There are duplicates due to problems last week with nebulous and with a crash on my computer. These need to be culled.
• isp
  • processing dates through cam (continuing). wrote a script to make this easier
  • trolling through database/images to add comment and other columns to isp database
  • trolling through database/images to check that isp raw exposures have 2 good copies. Currently 8% have bad or missing copies.
• gpc1/nebulous
  • investigating (with serge) bad raw images in nebulous/gpc1.
• grabbed files for larry
• helped pavlos some (need to help him more) to get ipp running.
• czar
  • restarted summitcopy (it had too many connections, causing problems for camera)
  • cleaned up lots of blood on thursday
• finished jtmd stacks (2011 and earlier)
• sick half a day

Roy Henderson

• sick on Thursday

• ippToPsps:
  • finally finished off code changes to load by box on the sky, configurable from the config. Still needs some tweaking.
  • finally implemented a class encapsulating the ippToPsps config file:
    • less code duplication as all classes can be passed config object and use 'getters' for the various fields
    • logger objects now created by config object, again, to reduce duplicated code across programs
    • a refresh() method means the config can be re-read at any time meaning config settings can be changed on-the-fly without a need to stop/start programs
    • config values can be printed to log on demand
    • re-factored all other code accordingly

• PSVO
  • fixed bug in PSVO reported by Nigel and made a release
  • slowly improving comments in PSVO code, generally neatening-up the code

• Other
  • czar Monday and Tuesday. Learned how to unstick LAP, which came in handy.
  • mailing list stuff - new list for psps working group
  • fixed bug in script that backs-up various databases - was not dumping contents, only schemas
  • updated PSPS news page after merge completed, also including IPP 3PI/OLD coverage for comparison
  • some cleanup after completed merge

Mark Huber

• ppSub DIFF_* stats and flags2 investigations with Larry and tracing out possible failure examples, revisiting filtering of detections from diffim residuals wih Ken Smith and monitoring cases of poor subtractions.
  • condition for neighbor and/or self-match to use source s/n rather than detection peak s/n
• updating IPP on macbook and local setup of photpipe for testing
• DRAVG minutes
• occasional kicking of LAP
• sick Friday

Bill Sweeney

• Completed testing of astrometric reference catalog for one of the STS fields. Ran 120 exposures using it. sigma_ra and sigma_dec are significantly improved: reduced by factor of about 8).
• Investigated by detectability queries for stack-stack diffs do not work. There are some significant problems with the method used to find the exposures.
• Queued and monitored processing of 162 M31 exposures in g, z, and y filters. Ran into a serious problem with some of the z band data. One one night the seeing was very good ~3 pixels and psphot's memory usage exploded causing machines to crash and jobs to fail. Tried running these on a machine with large swap space dedicated to these 5 chips. It took most of the weekend but the exposures completed. Need to investigate what psphot is doing. Also the data from 2009 did not come out well. There is something wrong with the recipes used for this data. Pattern correction?
• cleaned up database mess caused by ippc17 crashing repeatedly.
• one day vacation

Chris Waters