Changes between Initial Version and Version 1 of Mirroring

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

Mirroring

@@ +1 @@
+Also see the [http://groups.google.com/group/ps1net ps1net google group]
+
+= Questions/Notes: =
+
+# Server setup?
+# Performance of rsync on real-life updated dvo database tables? <br />This could be done using the Stripe82 data after successively updating dvo with more and more detections. May need to do this with the full stripe82 dataset, i.e. a few hundred square degrees, rather than the current 2.5x2.5 test set.
+# Transfer rate needed? Gene estimates: 
+## ~1 Mbit/s continuous for ''detections''
+## ~3 Mbit/s continuous for ''average tables'' ('''what about extended-source parameters?''')
+## ~75 Mbit/s continous for ''raw images'' (but might also want weights & masks - i.e. double the raw rate)
+Johannes Koppenhöfer has managed transfer rates of 400 Mbit/s from Hawaii to Garching, via two parallel rsync transfers at 200 Mbit/s each, but so far only over timescales of 10 minutes. The best sustained rate was ~80 Mbit/s over a few hours from ipp000 or alala to Heidelberg via rsync, though this was a much earlier experiment that should be repeated soon.
+
+== Data Transfer Requirements ==
+
+ * Case 1 : Total data volume based on IPP storage requirements:
+  * Raw Images: 
+   * 1 exposure per 90 seconds
+   * 11 hours per night
+   * 250 clear nights per year
+   * 3.5 years
+   * +10% for Detrend Data
+   * Total Raw Exposures = 423500 Exposures
+   * Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
+   * 2.2 bytes per pixel compressed
+   * 4800^2 x 60 = 1382400000 pixels = 3.0 GB / exposure
+   * Total Detrended Exposure Dataset = 1.27 PB
+  * Static Sky:
+   * 30000 square degrees
+   * 0.2 arcsec per pixel
+   * 10% overlap
+   * = 10 Terapixels
+   * 5 filters
+   * 10 bytes per pixel (uncompressed, 32bit floats) 
+   * 5 bytes per pixel (compressed)
+   * Total Static Sky Dataset = 250 TB
+   * 4 copies to be shipped (1 per year + 1 final)
+   * Total Static Sky = 1.0 PB
+  * Total Data Volume : 2.27 PB = 1.8x10^16 bits
+  * Time : 3.5 years = 1.1x10^8 sec
+  * Data Rate = 164 Mb/sec sustained
+
+ * Case 2 : Total data volume based on survey data size
+  * Raw Images: 
+   * 5500 3pi fields
+   * 60 exposures
+   * +10% for Detrend Data
+   * +25% for Other Survey Data
+   * Total Raw Exposures = 453750 Exposures
+   * Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
+   * 2.2 bytes per pixel compressed
+   * 4800^2 x 60 = 1382400000 pixels = 3.0 GB / exposure
+   * Total Detrended Exposure Dataset = 1.36 PB
+  * Static Sky (as above)
+   * Total Static Sky = 1.0 PB
+  * Total Data Volume : 2.36 PB = 1.9x10^16 bits
+  * Time : 3.5 years = 1.1x10^8 sec
+  * Data Rate = 173 Mb/sec sustained
+
+ * Case 3 : Only Raw (masked) survey data and static sky (ie, end users must re-detrend)
+  * Raw Images: 
+   * 5500 3pi fields
+   * 60 exposures
+   * +10% for Detrend Data
+   * +25% for Other Survey Data
+   * Total Raw Exposures = 453750 Exposures
+   * Detrended Images have 2 bytes per pixel uncompressed (signal)
+   * 1.0 bytes per pixel compressed
+   * 4800^2 x 60 = 1382400000 pixels = 1.38 GB / exposure
+   * Total Detrended Exposure Dataset = 626 TB
+  * Static Sky (as above)
+   * Total Static Sky = 1.0 PB
+  * Total Data Volume : 1.63 PB = 1.3x10^16 bits
+  * Time : 3.5 years = 1.1x10^8 sec
+  * Data Rate = 118 Mb/sec sustained
+
+== recommended TCP/IP settings (to be set in /etc/sysctl.conf): ==
+
+net.core.rmem_max=62500000
+
+net.core.wmem_max=62500000
+
+net.ipv4.tcp_rmem=4096 87380 62500000
+
+net.ipv4.tcp_wmem=4096 16384 62500000
+
+net.core.netdev_max_backlog = 2500
+
+net.ipv4.tcp_moderate_rcvbuf=1
+
+net.ipv4.tcp_timestamps=1
+
+net.ipv4.tcp_window_scaling=1
+
+net.ipv4.tcp_sack=1
+
+net.ipv4.tcp_ecn=1
+
+net.ipv4.tcp_adv_win_scale=7
+
+net.ipv4.tcp_syncookies=0
+
+net.ipv4.tcp_congestion_control=cubic
+
+
+== Speed test to Munich/Garching (single job, single file): ==
+
+# Hawaii->cluster: 224.21Mbit/s
+# Hawaii->MPE:      30.96Mbit/s
+# Hawaii->USM:      53.28Mbit/s
+
+
+== rsync command used for tests: ==
+
+\rsync -avP rsync://ipp0002.ifa.hawaii.edu/catdir.synth.grizy/n0000.tar.gz .