IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 9 and Version 10 of External_Supercomputing_Notes


Ignore:
Timestamp:
Dec 19, 2013, 12:25:27 PM (13 years ago)
Author:
watersc1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • External_Supercomputing_Notes

    v9 v10  
    106106
    107107It seems to me that any solution is going to require us to automatically ship data on some scale to LANL, automatically send a job to Moab, and automatically capture the result.  We should start on achieving those goal in a generic way rather than worrying about running IPP jobs specifically up front.
     108
     109== 2013-12-19 Telecon ==
     110
     111 * Storage.
     112  * Other local storage exists there, although it is significantly slower than transferring even from Hawaii.
     113  * The temporary scratch disk space is cleared automatically, although there is some leeway.  Files that are accessed (or updated?) have their retention time extended.  There are limits to this extension, so manually gaming the system is undesirable.
     114 * Network.
     115  * SVN access may be possible with exceptions in the firewall.  These exceptions will be required to point to only a small number of IP addresses.
     116  * Data transfer can be done through an intermediate node, although this is limited to only 20TB of temp space.  This suggests pushing (as listed above) or having a firewall hole to pull data through is a more workable solution.
     117  * [http://www.psc.edu/index.php/hpn-ssh HPN-SSH] exists, and can be used to speed up scp transfers (I'm not fully clear on the details, but the claim is up to 10x speed improvement).
     118 * We do have an account for moab processing, although by default our jobs should be "charged" to the correct one.  The allocation is sufficiently large that this isn't much of a concern.
     119 * Bundling tasks.
     120  * Due to the lack of database at the computing center, it's likely that we'll need to bundle a command list here, transfer that list along with the associated data, and then retrieve results.
     121  * Doing this with the SAS is probably the most appropriate first test.
     122 
     123== Bundling levels ==