Up: IPP at the ROE

Downloading Data with IPP

Ultimately, at the PS1 data distribution centre, there will be a data storage area allocated for us. This storage area will be populated (via symbolic links) with the data we require for our analysis. Bill Sweeney is in charge of the data distribution.

UPDATE: Our datastore is ​http://datastore.ipp.ifa.hawaii.edu/roe-ac-uk-md The rest of the wiki has been updated accordingly. -- EricTittley - 2009-11-17

On eday, we will maintain a mirror of the data in the data storage area.

But first, let's just learn how to set up basic downloads of data products.

Much of this follows: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/RunningTheDistributionClient which, except for the links, is pretty much dead on.

Setting up a local storage area and associated data base

Decide on a place to store the data

• set WORKDIR=/disk1/ps1/ert/IPP/PS1data
• mkdir $WORKDIR

Initialise the data base to hold the data metadata

• $SRCDIR/trunk/ippMonitor/scripts/dbadmin project localhost ipp PS1data
• pxadmin -dbname PS1data -create-mirror
• receivetool -dbname PS1data -definesource -source http://datastore.ipp.ifa.hawaii.edu -product roe-ac-uk-md -workdir $WORKDIR -comment "PS1 Data (local mirror)"

Ensure you can SSH to your local client

The pantasks controller needs to be able to ssh login-lessly to all nodes capable of processing data, including the local node.

Ensure it works.

• ssh localhost ls
• ssh hostname ls

SSH'ing to the local host is particularly problematic on systems with shared home directories (like the IfA) where the localhost entry in ~/.ssh/known_hosts can correspond to any machine on the network. It may need to be edited out.

Start download using pantasks

Start pantasks then issue the following commands

• module pantasks.pro
• module receive.pro
• add.database PS1data
• controller host add localhost
• run

For interest's sake, outside of pantasks you can examine info about the data base with:

• receivetool -dbname PS1data -list

What you get

You should get the directory

• $WORKDIR/gpc1/roe-ac-uk-md
  • gpc1 == Giga Pixel Camera 1
  • ps1-200910-2 is the project
  • The data is stored in the project directory.
• $WORKDIR/20091112
  • 20091112 is the date you run the download on, so will vary

-- EricTittley - 2009-11-17