Also see the ​ps1net google group

Questions/Notes:

1. Server setup?
2. Performance of rsync on real-life updated dvo database tables? <br />This could be done using the Stripe82 data after successively updating dvo with more and more detections. May need to do this with the full stripe82 dataset, i.e. a few hundred square degrees, rather than the current 2.5x2.5 test set.
3. Transfer rate needed? Gene estimates:
   • ~1 Mbit/s continuous for detections
   • ~3 Mbit/s continuous for average tables (what about extended-source parameters?)
   • ~75 Mbit/s continous for raw images (but might also want weights & masks - i.e. double the raw rate)

Johannes Koppenhöfer has managed transfer rates of 400 Mbit/s from Hawaii to Garching, via two parallel rsync transfers at 200 Mbit/s each, but so far only over timescales of 10 minutes. The best sustained rate was ~80 Mbit/s over a few hours from ipp000 or alala to Heidelberg via rsync, though this was a much earlier experiment that should be repeated soon.

Data Transfer Requirements

• Case 1 : Total data volume based on IPP storage requirements:
  • Raw Images:
    • 1 exposure per 90 seconds
    • 11 hours per night
    • 250 clear nights per year
    • 3.5 years
    • +10% for Detrend Data
    • Total Raw Exposures = 423500 Exposures
    • Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
    • 2.2 bytes per pixel compressed
    • 4800^{2 x 60 = 1382400000 pixels = 3.0 GB / exposure}
    • Total Detrended Exposure Dataset = 1.27 PB
  • Static Sky:
    • 30000 square degrees
    • 0.2 arcsec per pixel
    • 10% overlap
    • = 10 Terapixels
    • 5 filters
    • 10 bytes per pixel (uncompressed, 32bit floats)
    • 5 bytes per pixel (compressed)
    • Total Static Sky Dataset = 250 TB
    • 4 copies to be shipped (1 per year + 1 final)
    • Total Static Sky = 1.0 PB
  • Total Data Volume : 2.27 PB = 1.8x10^{16 bits}
  • Time : 3.5 years = 1.1x10^{8 sec}
  • Data Rate = 164 Mb/sec sustained

• Case 2 : Total data volume based on survey data size
  • Raw Images:
    • 5500 3pi fields
    • 60 exposures
    • +10% for Detrend Data
    • +25% for Other Survey Data
    • Total Raw Exposures = 453750 Exposures
    • Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
    • 2.2 bytes per pixel compressed
    • 4800^{2 x 60 = 1382400000 pixels = 3.0 GB / exposure}
    • Total Detrended Exposure Dataset = 1.36 PB
  • Static Sky (as above)
    • Total Static Sky = 1.0 PB
  • Total Data Volume : 2.36 PB = 1.9x10^{16 bits}
  • Time : 3.5 years = 1.1x10^{8 sec}
  • Data Rate = 173 Mb/sec sustained

• Case 3 : Only Raw (masked) survey data and static sky (ie, end users must re-detrend)
  • Raw Images:
    • 5500 3pi fields
    • 60 exposures
    • +10% for Detrend Data
    • +25% for Other Survey Data
    • Total Raw Exposures = 453750 Exposures
    • Detrended Images have 2 bytes per pixel uncompressed (signal)
    • 1.0 bytes per pixel compressed
    • 4800^{2 x 60 = 1382400000 pixels = 1.38 GB / exposure}
    • Total Detrended Exposure Dataset = 626 TB
  • Static Sky (as above)
    • Total Static Sky = 1.0 PB
  • Total Data Volume : 1.63 PB = 1.3x10^{16 bits}
  • Time : 3.5 years = 1.1x10^{8 sec}
  • Data Rate = 118 Mb/sec sustained

recommended TCP/IP settings (to be set in /etc/sysctl.conf):

net.core.rmem_max=62500000
net.core.wmem_max=62500000
net.core.netdev_max_backlog = 2500

net.ipv4.tcp_rmem=4096 87380 62500000
net.ipv4.tcp_wmem=4096 16384 62500000
net.ipv4.tcp_moderate_rcvbuf=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_ecn=1
net.ipv4.tcp_adv_win_scale=7
net.ipv4.tcp_syncookies=0
net.ipv4.tcp_congestion_control=cubic

Speed test to Munich/Garching (single job, single file):

  Hawaii->cluster: 224.21Mbit/s
  Hawaii->MPE:      30.96Mbit/s 
  Hawaii->USM:      53.28Mbit/s

rsync command used for tests:

\rsync -avP rsync://ipp0002.ifa.hawaii.edu/catdir.synth.grizy/n0000.tar.gz .