IPP Progress Report for the week 2010.12.06 - 2010.12.10

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• OC133 OSS data transfer is finished. ipp005 is now clean and relevant data are on mops01.

Warps: On the MOPS side, the warp files are located in /data/mops01.0/133.OSS/warps.oc133. They were individually md5-verified. On IPP side, the corresponding files were removed from ipp005 (precisely from /data/ipp005.0/heather/oss/warp, +1.4TB). The warps files are grouped in directories whose name is an exposure (e.g.: warps.oc133/o5500g0260o). Each exposure directory contains the skycells files derived from that exposure. For instance, warps.oc133/o5500g0260o is associated to 86 skycells (1058.069, [...], 1059.123, [...], 1148.024). There are 3 files for each skycell: cmf, fits, mask.fits (e.g. warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.cmf, warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.fits, warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.mask.fits).

Diffs: All files were transferred to /data/mops01.0/133.OSS/diffs.occ133. They were individually md5-verified. The diffs are grouped in directories which names are skycells (e.g. diffs.occ133/skycell.1148.003). Each skycell directory contains a varying number of files according to the diffs (but that number is multiple of 10). There are 5 files (cmf, inv.cmf, fits, mask.fits, inv.mask.fits) for both the unmagicked-nondestreaked and the magicked-destreaked diff (hence 5*2 = 10 files). Unmagicked-nondestreaked filenames contain the SR_ string while magicked-destreaked do not. For instance (from diffs.occ133/skycell.1148.003/), you can find the magicked-destreaked 551743718.gpc1:OC133.OSS.20101117:RINGS.V0:skycell.1148.003:RINGS.V0.skycell.1148.003.dif.93864.fits and the unmagicked-nondestreaked 555780169.gpc1:OC133.OSS.20101117:RINGS.V0:skycell.1148.003:SR_RINGS.V0.skycell.1148.003.dif.93864.fits. Note also that unfortunately the files do not have the numeric prefix (551743718 and 555780169).

I copied/pasted this e-mail in the /data/mops01.0/133.OSS/README file.

• Fixed publish scripts

• Database replication: nebulous is now replicated on ippdb02. But a mysql server can be the salve of only one master. It means that it is not possible to duplicate gpc1 and ippRequestServer on ippdb02.

• Helped Heather for static mask

Heather Flewelling

Roy Henderson

• PSPS
  • worked on a script which audits the current status of the loading to PSPS, i.e. what has loaded, what's merged, what didn't get processed etc.
  • of the exposures as yet unprocessed by ippToPsps, it generates a list of SMF files unavailable from Nebulous
  • started to process the 3351 exposures that failed first time around. So far so good (1288 successful, only 18 failures).
  • fixed (minor) Db corruption after numerous batches were marked as deleted when not (after datastore Db moved hosts)
  • PSPS VO
    • created a TableModel class in order to display SOAP result sets correctly in a table
    • made new query GUI component so users can query PSPS via SOAP then display results in dynamically sized table
    • ability to select range of values in a table to be plotted (not yet implemented)
    • casjobs errors are reported to a popup

• IPP
  • czar on Monday
  • czartool
    • added multiple constructors to Plotter class to make things cleaner
    • some work re-arranging schedule for czarpoll: now 'trimming' database after metrics are created
    • removed ability to stop/run servers; added link to metrics page
    • fixed bug where it was querying on 'new' instead of 'update' for update server

• other
  • some mailing list admin: removing bouncing emails
  • doctor all Wednesday afternoon

Bill Sweeney

Chris Waters

• Ran script to gzip compress all mdc files on disk. This appears to have freed roughly 25TB of disk space.
• Finished and tested realtime burntool code. Successfully burntools images at registration and automatically queues science exposures for chip->warp processing. Rewrote (but have not yet tested) the nightly science code to better handle the stack and diff stage processing, since it no longer needs to worry about chip.