Data Transfer Summaries and Experiences -- Release 2009.07.07

(Up to PS1 Science Processing Status (20090705)) (Up to IPP for PS1)

• CfA/Harvard: using IPP data distribution protocol with a local python client on Odyssey, 1.4TB. in about 1 day. The limiting factor has been issues with the receiving disk.

• JHU: using the IPP data distribution client, obtained ~1TB in a couple days (more details to come).

• Garching: 2TB in less than 2 days using the IPP data distribution client.

• Edinburgh: awaiting delivery of 20TB of disk.

• Durham: 290 GB in about 1.5 days, or 2Mbytes/sec, using a recursive wget (no IPP client, no python script).

---

WARNINGS from Bill 2009 July 14

• Some runs have been inadvertently posted to the data store multiple times. Some errors at import time are to be expected.

• The receive clien'ts script receive_file.pl isn't very careful with regard to cleaning up temporary files. Some people have had problems running out of space.

• There were some errors in the code with regard to reporting errors related to filesets. Update ippScripts/scripts/receive_fileset.pl

• There was a bug in the workflow for filesets that could cause errors to be reported. Fixes are in ippTools/share/receivetool_pendingfilest.sql and ippTools/share/receivetool_pendingfile.sql