Changes between Version 29 and Version 30 of PS1_IPP_Czarlog_20140224

Timestamp:

Mar 2, 2014, 12:46:55 PM (12 years ago)

Author:

eugene

Comment:

--

PS1_IPP_Czarlog_20140224

@@ -59 +59 @@
 === Sunday : 2014-03-02 ===
  * 11:00 EAM : I am stopping all processing to do a major revamp on the way we allocate hosts.  more in a moment...
+ * 12:45 EAM : 
+{{{
+last week, we were having some trouble keeping up with nightly science.  we did not have enough machines at night to support the necessary exposure rate.  when I tried to make adjustments, I decided it was difficult to see how best to balance staticsky and stdscience with the machines organized as they were.  therefore, I have made a significant change to the way the pantasks hosts are allocated.
+
+one source of confusion has been that not all 'wave1' or 'compute3', etc, machines are alike or even very similar.  to avoid overloading some in the same class, we have been either underloading others or going through some silly gymnastics.
+
+I have re-sorted the list of machines into groups by compute and storage based on their total memory (and at bit on the number of cores).  The hosts are now in the following groups:
+
+storage: 
+s0 : 16GB or 20GB + 8 cores
+s1 : 24GB + 8 cores
+s2 : 32GB + 8 cores
+s3 : 40GB or 48GB + 12 cores
+
+compute:
+c0 : 24GB + 8 or 12 cores 
+c1 : 32GB + 8 cores
+c2 : 48GB + 12 cores
+
+(c1 is split for now into c1a and c1b, where c1a are the apache servers)
+
+Next, I looked at the typical usage for stdscience and our bottom-line requirement there.  For stdscience, most of the jobs take a modest amount of memory (~2GB) and do not normally keep all 4 threads spinning (say 2 on average).  Meanwhile, the typical processing time in stdscience is ~3.85 node-hours per exposure total with (chip,cam,fake,warp,diff) = (1.5,0.12,0.06,1.04,1.15).  For a solid NASA night, we get about 700 exposures.  If we process 70 exposures per hour, we can keep up with the night (done in 10 hours).  That says we need a total of ~270 jobs active.  
+
+I've assigned stdscience to the storage nodes only with the following number of jobs per host, based on the memory and cpu usage:
+s0 : 3
+s1 : 4
+s2 : 5
+s3 : 6
+
+That gives us a total of 271 active connections, so we should be OK in terms of nightly science processing with that level.
+
+Next, I've assigned only a modest number of machines to stack: only 2 x c1a, or about 16 jobs.  We are not doing too much stacking at the moment.  when stack gets behind, we can swap stdscience and stack allocations during the day.
+
+That leaves c0, c1b, and c2 machines for staticsky.  that gives us ~130 staticsky jobs.  
+
+For the other pantasks, I've given a modest allocation on either s0-s3 (cleanup, detrend, registration, summitcopy, distribution) or c2 (pstamp & publish).  
+
+Note that I have removed the deepstack list for now : we will need to re-vamp it when the new machines are ready, and we will have to juggle the stare and c2 nodes with ipptopsps and staticsky.
+
+Sifan and Haydn have set up a couple of new (actually old) machines at MTRC-B.  These are low-cpu, low-mem machines which should be sufficient for both apache and pantasks servers. Over the next few days, I've like to test them and get the services used by some of the compute nodes moved over there.  that will let us move fully use the compute hardware.
+
+}}}
+