| | 1 | 2008.01.25 : EAM |
| | 2 | |
| | 3 | I am downloading a large amount of megacam archival data to the IPP cluster at MHPCC. There are roughly 36TB of compressed raw image data in the Megacam archive at the moment, though some fraction of that is proprietary. I have set up a set of jobs to download all public data. I am using the following scripts: |
| | 4 | |
| | 5 | ~eugene/src/bin/shell/grabmegacam.pl : takes the name of an index file, and optionally an OBS number range |
| | 6 | ~eugene/src/bin/shell/get.megacam.year : downloads all images for a year (02, 03, etc) into directories by megacam run ID. it uses a set of tables in |
| | 7 | ~eugene/catdir/megacam, generated from the elixir image registration tables. |
| | 8 | |
| | 9 | The following IPP nodes will hold megacam data for this experiment: |
| | 10 | |
| | 11 | /data/ipp016.0 : 2005 |
| | 12 | /data/ipp017.0 : 2003 |
| | 13 | /data/ipp018.0 : 2004 |
| | 14 | /data/ipp019.0 : 2002 (also has some test gpc1 data and the old user home directories) |
| | 15 | /data/ipp020.0 : 2006 |
| | 16 | /data/ipp021.0 : 2007 |
| | 17 | |
| | 18 | we also currently have: |
| | 19 | |
| | 20 | /data/ipp007.0 : empty |
| | 21 | /data/ipp008.0 : empty |
| | 22 | /data/ipp011.0 : empty |
| | 23 | /data/ipp020.0 : empty |
| | 24 | /data/ipp021.0 : empty |
| | 25 | |
| | 26 | /data/ipp004.0 : home, bill's megacam download |
| | 27 | /data/ipp005.0 : gpc1 stare data, bill's megacam download |
| | 28 | /data/ipp006.0 : josh's gpc1 download tests |
| | 29 | /data/ipp009.0 : josh's gpc1 download tests, allskycam downloads |
| | 30 | /data/ipp010.0 : josh's gpc1 download tests, isp, allskycam downloads |
| | 31 | |
| | 32 | |
| | 33 | * data downloaded to date (2/20) |
| | 34 | |
| | 35 | The following IPP nodes will hold megacam data for this experiment: |
| | 36 | |
| | 37 | /data/ipp016.0 : 2005 -- 19855 of 20634 images; none fits-compressed |
| | 38 | /data/ipp017.0 : 2003 -- 17332 of 17383 images; (03Am03, 04, 03Bm01, 02), 03, 04, 05 not fits-compressed |
| | 39 | /data/ipp018.0 : 2004 -- 17214 of 17390 images; (04Am06, 04Am07, 04Bm02, 03), 04, 05, 06, 07 not fits-compressed |
| | 40 | /data/ipp019.0 : 2002 -- 2901 or 2908 images, all fits-compressed |
| | 41 | /data/ipp020.0 : 2006 -- not yet downloaded |
| | 42 | /data/ipp021.0 : 2007 -- not yet downloaded. |
| | 43 | |
| | 44 | * registering: |
| | 45 | |
| | 46 | ipp016: foreach dir ( 05* ) |
| | 47 | foreach? ipp_serial_inject.pl --telescope cfht --camera megacam --workdir /data/ipp016.0/megacam/$dir.work --end_stage reg --dbname megacam $dir/*.fits |
| | 48 | foreach? end |
