Datastore Sky Tessellation

There is a desire to organize the datastores for PV3 into sky chunks that are of a similar size of data, organized by dxlayer and load merge machine. The intent is to always feed the PSPS a chunk of batches that add up to a similar size in bytes for each PSPS machine so there are no surprises. It also follows the new PSPS hardware design including the switches and network. See ​PSPS Hardware Plan C Over the course of the ingest and merge, the only thing that will significantly vary as we ingest across the sky is the cumulative size of the merged database. Each subsequent addition prior to merge, will be about the same size, and will be handled by a similar machine. Our greatest uncertainty at this stage is the scaling with the total size, not the scaling per chunk. This page describes a "datastore tessellation".

Step 1

Subdivide sky into the 32 declination layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data. These layers correspond to one each of the new PSPS Load/Merge machines. The layers have a minimum height greater than half the field of view. This is so that no single frame image spans more than 2 layers, hence each layer must be >= 1.7 degrees. The spacing will be calculated from the full data set - it is more complex than just the density of sources on the sky - there is also the difference in the tables populated in the division between galactic and extragalactic regions. NOTE the PSPS convention is the northernmost layer, including the CNP, is Layer 01, i.e. they count down from the north pole. This is an odd convention, but we should stick with it because eventually we will expand further south, and will not want to relabel everything, so we can just add more layers with higher index numbers.

Step 2

There are 8 DXlayer machines, each dxlayer machine is assigned to 4 contiguous Declination layers. Alternatively can think of this as the 8 DXlayers each being subdivided into 4 Dec Layers making the 32 in total. To aid in the convenience of making batches, the boundaries between these layers will be round to the nearest integer degree Declination. NOTE distinction be DXLayer and Dec Layer. There are 4 DecLayers per DXLayer. (this was not the original source of the DXLayer term, but it works).

Step 3

For each set of 4 layers, called dxlayer[num], subdivide those to have SLICES of equal amount of data. This example here only has 8 SLICES per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees). The division should be made by dividing the data in Right Ascension along a layer into X SLICES of approximately equal data size, where X is 24 for the Forced Warp and Detections. These boundaries between SLICES are also rounded off to the nearest integer degree of Right Ascension.

Step 4

Each of these rectangular in (RA,DEC) CHUNKS corresponds to a Datastore. Each datastore is of the same size. (note - i think better to for DS label to have a (x,y), because that is how we will load and merge. i.e. DS01-1 is most north and starts at RA >= 0. A SLICE is a sequential set of chunks in increasing RA, with a single chunk from each DXLAYER. ) labels in plots need to be adjusted. We will then Load by SLICE. i.e the first 8 datastores cover a complete vertical region covering from Dec 90 to -30 8 chunks of batches. The SLICES will eventually be mismatched horizontally because of the density of sources, but that is fine. After a complete SLICE with 8 DXLAYERS has Loaded we will MERGE. So the number of Merges will by definition be either 6, 12 or 24 depending on batch type as per the table below. The slices may need to be STAiRCASED in integer degrees to keep the MERGE efficient. The MERGE time will be dominated by the slowest merge of the 32 Dec Layers - hence the reason we will want to staircase it pretty closely. Either that or define the slices as precisely as the Dec Layers, and modify the batch making code.

Final Construction

For OB/FO/DO/DF, because these are small batch sizes, we will subdivide each dxlayer[num] into 6 segments, giving a total of 48 datastores. ST/FG will have more than Object/Diff types of batches, so will divide each dxlayer[num] into 12 segments, giving a total of 96 datastores. P2/FW are the heaviest, and will be subdivided into 24 segments per dxlayer, for a total of 192 datastores.

The goal is to have PSPS load 8 datastores per merge (or possibly a multiple of that).

The names of the datastores will follow this format: PSPS_PV3_[batch]_dx[dxlayer]_seg[num], an example would be PSPS_PV3_OB_dx1_01

Expectations for the different batch types

type	dxlayers	slices per dxlayer	total datastores	load order	approx number	approx size
OB	8	6	48	1st
ST	8	12	96	2nd
P2	8	24	192	4th
FW	8	24	192	3rd
FO	8	6	48	1st
FG	8	12	96	2nd
DO	8	6	48	5th
DF	8	6	48	5th

After a "load order" group has been completely (all sky) merged and subsequently copied to the Hot Slice/Head, then there will be a SQLServer dump/copy to STScI. Each dump/copy will write over the former.

• a note that may as well go here, eventually we will have the Derived Data Products like photo, extinction, proper motion,and those will be another batch type, although small.