Cluster Storage Notes

Accounting

2010-09-27	Item	Balance
Total Disk Space (1k blocks) :	762,480,098,880	762,480,098,880
Contained in Nebulous:	721,827,135,488	40,652,963,392
Identified Not in Nebulous:	66,723,319,220	-26,070,355,828

Accounting

2010-09-20	Item	Balance
Total Disk Space (1k blocks) :	762,485,617,760	762,485,617,760
Contained in Nebulous:	690,849,054,720	71,636,563,040
Identified Not in Nebulous:	62,557,092,220	9,079,470,820

In Nebulous By Host

Usage by filetype/processing stage in gigabytes. (2010-09-27)

Host	BTABLE	BTPIX	CAM	CHIP	CMF	DETREND	DIFF	JPEG	LOG	MDC	MEGACAM	OTA	PSF	SKYCELL	STACK	STATS	TRACE	UNKFITS	UNKNOTFITS	WARP	nebulous	total_used	zdisk
ipp004.0	1.77	0.00	52.10	6.84	16.64	13.92	3.37	8.59	8.47	98.37	0.00	1524.07	6.42	3.30	10.36	0.42	0.37	43.17	49.71	1322.36	3170.27	9232.76	10158.16
ipp005.0	17.69	0.00	340.19	426.64	53.16	140.01	593.09	24.89	36.96	177.53	0.00	8898.45	13.24	31.31	186.72	1.39	0.01	169.13	235.13	267.09	11612.62	13566.52	40916.94
ipp006.0	22.17	0.00	275.02	677.92	106.85	184.45	796.31	46.66	63.72	346.43	0.00	7737.29	24.58	41.40	296.53	1.78	0.09	140.59	170.52	517.53	11449.85	16015.41	40916.94
ipp007.0	15.38	0.00	201.79	591.79	94.36	155.12	725.10	40.84	53.45	300.99	0.00	5519.18	21.67	36.54	274.04	1.49	0.08	105.12	125.48	456.91	8719.34	10966.35	40916.94
ipp008.0	12.97	0.00	277.24	315.07	53.97	130.46	496.41	25.14	41.08	190.01	0.00	6025.76	13.61	27.66	145.94	1.27	0.00	130.49	194.72	226.61	8308.42	9344.27	10158.16
ipp009.0	12.82	0.00	148.00	396.60	99.56	164.45	354.36	38.46	58.03	313.96	10.20	6146.71	22.19	27.89	143.18	1.70	0.08	84.33	105.91	274.50	8402.93	9341.77	10158.16
ipp010.0	13.91	0.00	160.71	394.46	87.51	169.59	440.16	40.81	59.85	318.48	0.00	6364.01	22.73	31.46	168.63	1.74	0.08	81.05	110.29	310.15	8775.62	9339.91	10158.16
ipp011.0	15.37	0.00	192.90	381.25	97.70	169.70	368.90	42.88	56.19	346.22	0.00	6363.16	23.95	33.82	184.74	1.70	0.09	105.90	128.98	315.06	8828.52	9340.25	10158.15
ipp012.0	10.35	0.00	73.54	154.66	67.99	94.95	128.73	35.71	34.19	235.40	5.82	4768.52	16.60	24.17	263.44	0.82	0.02	47.28	45.33	224.05	6231.57	6830.15	10158.15
ipp013.0	13.85	20.59	144.97	396.93	92.42	160.06	483.66	46.58	51.79	329.33	0.00	6184.26	24.10	33.81	283.74	1.64	0.01	96.15	107.54	249.80	8721.21	9341.04	10158.15
ipp014.0	2.55	697.81	13.91	288.96	13.42	882.13	72.83	2.16	13.82	47.86	0.00	2885.05	2.62	1.13	1.63	1.18	0.32	11.19	5.88	50.55	4995.01	5463.32	10118.31
ipp015.0	10.58	229.91	152.33	504.96	100.77	168.72	424.99	27.49	82.65	312.86	0.00	6276.29	24.27	19.64	159.12	3.24	0.48	51.18	121.45	202.60	8873.51	9526.20	10158.15
ipp016.0	14.70	0.00	210.52	285.52	62.11	133.07	357.74	27.83	47.49	216.33	0.00	7066.67	15.59	19.74	90.40	1.47	0.01	135.79	169.24	163.22	9017.44	9576.27	10158.15
ipp017.0	10.58	0.00	159.25	380.04	98.20	175.00	453.91	44.70	55.62	324.66	0.00	6278.33	23.98	32.16	278.91	1.71	0.01	99.20	107.56	276.12	8799.96	9340.26	10158.15
ipp018.0	7.94	0.00	175.07	415.12	66.01	208.92	442.57	28.59	33.27	216.40	0.00	6704.23	17.01	21.25	154.41	1.33	0.06	99.25	129.32	188.53	8909.29	9340.32	10158.15
ipp019.0	15.09	0.00	134.50	370.31	86.65	136.86	495.00	39.67	52.98	298.79	0.00	5134.32	21.71	28.33	211.57	1.33	0.01	89.80	90.40	237.00	7444.32	9339.95	10158.15
ipp021.0	15.23	0.00	175.79	351.69	93.82	173.11	371.13	39.66	58.51	336.77	0.00	6424.23	23.76	31.08	159.74	1.73	0.09	101.04	117.10	258.56	8733.02	9220.67	10158.15
ipp023.0	19.69	242.70	682.70	472.23	172.57	932.62	782.04	71.46	173.20	587.39	0.00	11422.37	43.10	38.06	263.11	5.51	0.30	385.46	403.19	442.50	17140.18	19615.73	20427.84
ipp024.0	23.71	254.08	375.63	672.29	216.41	278.42	833.80	67.45	181.79	659.80	11.72	14209.09	47.27	44.25	320.11	5.68	0.32	146.30	243.73	478.67	19070.55	19616.33	20427.84
ipp025.0	26.97	0.00	625.67	901.15	162.49	269.89	1055.50	67.30	151.40	721.06	0.00	13239.71	49.35	59.37	376.14	5.32	0.24	245.35	435.60	704.32	19096.82	19617.77	20427.84
ipp026.0	18.62	245.48	376.90	466.80	194.06	268.06	963.22	77.02	196.70	734.20	0.00	14204.17	52.09	45.79	387.36	5.96	0.99	165.29	270.80	427.54	19101.05	19617.29	20427.84
ipp027.0	20.24	0.00	437.73	476.49	116.68	147.11	1105.54	65.15	55.36	434.00	0.00	9509.60	31.57	58.34	389.25	2.04	0.01	238.62	306.44	483.48	13877.66	15789.76	20427.62
ipp028.0	25.13	325.99	375.82	680.27	245.28	274.67	1015.88	75.88	215.03	719.14	0.00	13578.60	50.70	46.61	377.00	6.25	2.68	155.89	279.25	563.24	19013.31	19619.08	20427.84
ipp029.0	25.72	236.04	356.44	696.17	232.48	290.66	1007.15	74.03	201.43	671.54	0.00	13728.28	48.70	44.36	375.87	5.82	0.34	156.71	230.58	466.74	18849.02	19618.98	20427.84
ipp030.0	19.94	240.18	293.28	564.87	195.94	382.81	971.90	61.87	167.26	573.18	1889.71	11747.98	43.23	37.40	337.38	5.24	0.27	140.13	200.74	433.81	18307.12	19622.53	20427.84
ipp031.0	27.22	253.29	399.90	688.53	232.48	299.14	1039.03	73.20	210.17	691.33	0.00	13893.74	49.89	44.16	370.90	5.90	0.34	169.56	243.28	507.61	19199.68	19642.19	20427.84
ipp032.0	26.01	240.75	374.89	1041.47	245.86	352.22	971.46	66.37	214.87	717.28	0.00	13576.94	51.73	35.82	263.05	6.54	0.59	162.97	280.50	487.16	19116.48	19678.74	20427.84
ipp033.0	29.20	323.95	393.33	1047.23	245.15	351.67	883.79	62.93	216.22	712.43	0.00	13432.88	51.44	35.47	226.85	6.62	2.31	163.80	247.10	497.59	18929.97	19660.12	20427.84
ipp034.0	23.09	268.76	371.74	1021.44	254.48	360.06	855.95	68.35	213.55	721.55	0.00	13390.99	52.34	40.71	298.19	6.49	2.51	157.13	301.00	523.13	18931.46	19675.53	20427.84
ipp035.0	25.16	213.45	323.28	983.11	236.69	354.75	841.31	66.71	200.06	668.27	0.00	11711.93	49.66	35.54	300.47	6.17	0.36	127.25	218.66	420.76	16783.60	19681.14	20427.84
ipp036.0	22.02	259.77	357.79	850.35	224.01	333.54	848.97	68.53	201.92	701.98	0.00	12821.62	51.69	38.57	329.21	6.37	0.41	137.78	250.26	427.48	17932.26	19664.81	20427.84
ipp037.0	15.00	0.00	308.10	369.10	56.17	135.70	896.47	24.10	31.72	182.03	0.00	6745.05	13.09	30.63	183.92	1.18	0.01	164.17	213.32	429.50	9799.26	12781.61	20427.84
ipp038.0	21.62	92.64	426.06	689.77	239.08	341.16	941.59	68.68	203.70	588.73	2004.66	11784.62	44.47	39.34	352.92	4.73	0.03	156.45	262.39	503.84	18766.50	19643.61	20427.62
ipp039.0	20.09	130.24	487.85	627.70	229.21	247.62	976.09	87.29	239.90	687.23	9.48	13750.32	51.22	48.37	407.64	5.33	0.04	179.60	329.17	488.72	19003.12	19624.07	20427.62
ipp040.0	13.91	21.21	224.79	622.05	182.07	293.77	903.31	49.67	138.07	503.37	1321.22	7466.59	38.68	31.44	326.91	3.96	0.01	77.17	141.46	460.67	12820.32	18687.74	19496.31
ipp041.0	21.13	90.42	309.88	696.67	208.11	210.01	965.23	55.50	153.25	513.64	0.00	14732.01	38.51	35.88	328.39	4.07	0.02	119.21	183.69	439.36	19104.97	19620.16	20427.62
ipp042.0	22.62	74.87	423.57	750.21	251.59	343.51	1024.74	83.86	187.16	657.62	2087.51	11379.72	48.50	56.44	441.96	4.36	0.01	165.25	245.43	565.94	18814.89	19618.76	20427.62
ipp043.0	19.96	35.14	354.59	747.09	231.57	363.71	951.37	60.88	168.37	553.23	2003.12	11477.34	41.58	37.97	306.39	4.40	0.02	160.52	229.88	457.83	18204.99	19618.98	20427.62
ipp044.0	24.10	100.83	383.88	693.73	213.12	349.40	888.33	57.12	168.24	495.31	1983.47	12305.22	37.11	35.14	237.20	3.92	0.01	175.30	247.95	441.41	18840.79	19618.18	20427.62
ipp045.0	17.80	92.43	338.90	298.61	202.11	287.62	233.47	48.32	154.69	468.36	1979.21	11145.86	35.35	27.68	137.17	4.01	0.01	99.49	187.03	171.50	15929.64	16601.39	20427.62
ipp046.0	24.44	96.65	384.70	716.59	237.19	358.23	829.40	62.60	194.02	567.46	1957.10	12121.21	42.91	36.41	253.21	4.57	0.01	141.46	239.77	454.61	18722.54	19619.97	20427.62
ipp047.0	26.85	99.09	414.15	1114.27	294.85	309.07	851.70	72.90	327.57	747.64	601.82	12718.10	57.57	45.70	312.83	6.52	0.01	116.26	291.20	527.51	18935.64	19649.09	20427.62
ipp048.0	29.50	25.64	489.01	1019.68	250.58	292.19	859.77	70.27	196.93	649.15	12.08	13625.62	50.05	41.38	327.86	5.47	0.01	203.92	319.07	419.32	18887.52	19643.50	20427.62
ipp049.0	23.77	27.68	474.40	1027.73	207.12	229.43	935.38	60.54	127.41	547.44	0.00	13729.27	42.70	42.89	281.08	4.68	0.01	185.19	334.10	412.69	18693.53	19652.63	20427.62
ipp050.0	19.59	104.02	374.98	874.21	220.30	254.95	949.78	65.00	179.36	621.68	241.83	10825.90	48.27	35.99	316.13	5.21	0.01	139.62	228.27	450.99	15956.11	19661.69	20427.62
ipp051.0	28.69	131.70	414.79	1037.65	247.94	282.48	938.45	69.28	191.11	634.13	312.76	12461.79	48.66	39.44	355.48	5.32	0.01	185.72	268.23	527.60	18181.22	19707.85	20427.62
ipp052.0	26.99	124.88	520.03	927.61	235.83	286.07	867.45	70.75	205.15	659.17	247.64	13577.11	50.33	42.19	337.30	5.60	0.02	201.43	324.38	456.30	19166.22	19621.93	20427.62
ipp053.0	16.98	0.34	349.92	610.82	113.37	172.19	955.26	37.48	69.35	274.37	0.00	8461.24	20.88	37.27	224.92	1.84	0.00	178.70	240.11	453.74	12218.78	18893.65	20427.62
Host	BTABLE	BTPIX	CAM	CHIP	CMF	DETREND	DIFF	JPEG	LOG	MDC	MEGACAM	OTA	PSF	SKYCELL	STACK	STATS	TRACE	UNKFITS	UNKNOTFITS	WARP	nebulous	total_used	zdisk
nebulous	918.70	5300.54	15312.53	29724.67	7885.91	12913.28	35151.58	2571.17	6343.02	23074.12	16679.36	479075.42	1700.70	1713.32	12759.33	181.03	13.74	6792.38	10211.12	20066.18	688388	0.00	0.00
Host	BTABLE	BTPIX	CAM	CHIP	CMF	DETREND	DIFF	JPEG	LOG	MDC	MEGACAM	OTA	PSF	SKYCELL	STACK	STATS	TRACE	UNKFITS	UNKNOTFITS	WARP	nebulous	total_used	zdisk
2010-09-17	881.64	5300.54	14614.97	24881.73	7670.06	10169.85	31880.33	2527.01	6235.41	22456.17	16679.36	466606.25	1653.25	1666.95	10931.73	175.16	13.80	6554.09	9737.67	18208.93	658845	0.00	0.00
2010-08-31	700.15	5043.63	12153.41	21797.65	6646.25	8173.74	102149.79	2215.34	5484.88	19422.63	16118.97	412240.75	1428.41	1417.45	9248.43	152.88	13.75	5045.70	8066.48	17265.02	0.00	0.00	0.00
2010-06-30	632.91	5300.54	10107.98	50342.60	5812.37	8825.46	138862.67	1903.96	5105.31	16518.10	16679.36	389417.69	1222.54	1090.20	7452.35	132.42	13.81	3360.59	6405.75	33418.52	0.00	0.00	0.00
2010-03-10	434.96	5300.54	5097.21	47390.77	3482.93	7290.61	79614.43	1038.09	3789.50	8553.75	16679.36	371492.33	559.41	466.39	3028.16	82.57	13.64	1994.03	2681.70	88722.46	0.00	0.00	0.00
2010-01-28	407.88	27232.79	3494.31	33465.69	2862.11	44719.29	59782.64	601.59	2854.32	5212.26	16679.36	337526.94	329.86	174.01	1851.94	62.14	13.69	1357.02	1575.63	59263.81	0.00	0.00	0.00
2009-12-01	271.22	27232.79	2602.76	53268.00	2094.33	34487.15	78070.53	454.34	2124.31	3898.29	16510.10	279834.80	229.40	119.03	671.66	48.95	13.48	909.36	1106.52	76556.49	580503.53	0.00	0.00

By Host

Check of the disk space currently (2010/09/17) being used by data not stored in nebulous. These data take up about X% of the total space (X% of the used space).

Host	Total Space (1K-blocks)	Used	Free	Non-nebulous	% Free	% Non-neb
ipp004	10651598144	9668172736	983425408	6225095300	9	58
ipp005	42904518272	12735045568	30169472704	1849334452	70	4
ipp006	42904518272	15012928864	27891589408	4352248276	65	10
ipp007	42904518272	8645114336	34259403936	843846780	80	2
ipp008	10651598144	8869014528	1782583616	880416552	17	8
ipp009	10651598144	8893309504	1758288640	783683760	17	7
ipp010	10651598144	8897620224	1753977920	387961520	16	4
ipp011	10651588352	8894690688	1756897664	319028872	16	3
ipp012	10651588352	5976266880	4675321472	385767604	44	4
ipp013	10651588352	8931181152	1720407200	421814280	16	4
ipp014	10609814272	5720860416	4888953856	430256580	46	4
ipp015	10651588352	9621174208	1030414144	520384336	10	5
ipp016	10651588352	9836861824	814726528	417492024	8	4
ipp017	10651588352	8874684384	1776903968	354581528	17	3
ipp018	10651588352	9254215360	1397372992	272969584	13	3
ipp019	10651588352	8943130784	1708457568	1744913380	16	16
ipp020	10651588352	8928654080	1722934272	162515732	16	2
ipp021	10651588352	8887911328	1763677024	275423028	17	3
ipp023	21420144640	19607102880	1813041760	2380961720	8	11
ipp024	21420144640	19606903968	1813240672	384515872	8	2
ipp025	21420144640	18882022880	2538121760	385165600	12	2
ipp026	21420144640	19606763456	1813381184	353597732	8	2
ipp027	21419913728	14865771520	6554142208	1685877568	31	8
ipp028	21420144640	19606896736	1813247904	435620552	8	2
ipp029	21420144640	19606945408	1813199232	607774312	8	3
ipp030	21420144640	19609195840	1810948800	1137047924	8	5
ipp031	21420144640	19606762688	1813381952	333630472	8	2
ipp032	21420144640	19608620544	1811524096	398919620	8	2
ipp033	21420144640	19608218464	1811926176	561960912	8	3
ipp034	21420144640	19609287296	1810857344	572208144	8	3
ipp035	21420144640	19621322912	1798821728	2809078900	8	13
ipp036	21420144640	19609280288	1810864352	1578484576	8	7
ipp037	21420144768	11684774304	9735370464	2819297236	45	13
ipp038	21419913728	19659111424	1760802304	704189768	8	3
ipp039	21419913728	19607913472	1812000256	431892560	8	2
ipp040	20443361792	18889056832	1554304960	5962392016	8	29
ipp041	21419913728	19605182176	1814731552	333110512	8	2
ipp042	21419913728	19548094432	1871819296	663995860	9	3
ipp043	21419913728	19606154016	1813759712	1319162548	8	6
ipp044	21419913728	19883039680	1536874048	646543760	7	3
ipp045	21419913728	17349764416	4070149312	633036376	19	3
ipp046	21419913728	19609665056	1810248672	747506716	8	3
ipp047	21419913728	19611711584	1808202144	537633340	8	3
ipp048	21419913728	19610930432	1808983296	588417472	8	3
ipp049	21419913728	19610054208	1809859520	803173324	8	4
ipp050	21419913728	19769048960	1650864768	3697905340	8	17
ipp051	21419913728	19671217344	1748696384	1398066436	8	7
ipp052	21419913728	19606953216	1812960512	333346980	8	2
ipp053	21419913728	18057996832	3361916896	6684844484	16	31

---

By Keyword

Keyword matching done based on summing the disk usage of all directories with the name /data/@HOST@/keyword. (2010-09-17)

Keyword	Total Space Used	% of cluster
gpc1_distribution	15201737384	2
gpc1	12023701776	1
gpc1_destreak	5920872104	1
ipp018.0	5414586080	1
beaumont	4259782908	0
sydney	3466959788	0
ipp	2904118824	0
gpc1_destreak_backup	1873868376	0
home.OLD	1501499432	0
heather	1493126276	0
backup	1272385328	0
home	1265241388	0
price	1035358932	0
eam_tmpdir	827888672	0
ipp007.0	787373580	0
iqanalysis	715451824	0
rodney	433771656	0
gpc1dark	317590620	0
MOPS	304913816	0
bills	295415636	0
eugene	228777276	0
ipp006.0	188832180	0
billsstuff	179740868	0
rhenders	146636364	0
ipp005.0	139375352	0
watersc1	93108736	0
old	90156472	0
tmp	64369448	0
isp	41834288	0
allskycam	21680528	0
kaiser	19740236	0
jt	7146200	0
gpc1_dist	6798048	0
datastore	4508376	0
burntool_logs	3709052	0
md09smf	2228540	0
eamtest	1683728	0
pstamp-work	323632	0
pstamp	305600	0
megacam	256016	0
rsync.back.log	120360	0
ippMonitor	106240	0
rsync.back.v2.log	2444	0
gpc1_rel_200901	2332	0
giebink	2060	0
gpc1-magic	1244	0
mysql	748	0
jhoblitt	396	0
rsync.back.v3.log	288	0
lost+found	240	0
serge	236	0
ippTests	236	0
MD08.200906.v0_0	36	0
mkhome.sh	4	0
sizes.dat	4	0
pcontrol.log	4	0
node_update.sh	4	0
rsync.pro	4	0
heather.zagursky	0	0
foo	0	0
schastel	0	0
disktest	0	0

Orphan / Widow Counts

Counts of nebulous orphans (disk files with no matching nebulous database instance) and widows (nebulous database instances with no matching disk files). Pre-demo month values: 2010-01-29

	2010-01-29	2010-01-29	2010-01-29	2010-03-04		2010-03-04	2010-03-04
Host	Orphans	Number Zero-size	Widows	Orphans		Number Zero-size	Widows
ipp004.0	952	156	18	274		145	24
ipp005.0	1250	94	0	1629		114	94
ipp006.0	1177	260	2	1076		252	22
ipp007.0	1067	213	11	1285		219	18
ipp009.0	4199	1654	6	2912		1614	91
ipp010.0	1880	160	2	2333		183	32
ipp011.0	3454	474	12	1491		419	94
ipp012.0	1939	127	182	620		95	12
ipp013.0	581	102	19	509		95	47
ipp014.0	52768	386	0	52871		400	20464
ipp015.0	591	39	1	1605		143	54
ipp016.0	1418	287	2	588		287	35
ipp017.0	2271	297	396	2213		301	126
ipp019.0	837	119	5	304		83	13
ipp020.0	934	244	4	530		230	50
ipp021.0	2814	246	53	2549		226	248
ipp023.0	7619	124	23	7210		116	1023
ipp024.0	3301	1034	4	6642		1234	3993
ipp025.0	13688	12304	9	13704		12327	288
ipp026.0	1600	729	30	4201		928	71
ipp028.0	12535	11356	140	26221		12131	3340
ipp029.0	1514	969	16	12334		1547	566
ipp030.0	1050	20	10	8267		424	141
ipp031.0	842	105	10	5598		360	4314
ipp032.0	782	480	6	7075		802	306
ipp033.0	4973	4206	48	12396		4591	48
ipp034.0	2202	2135	31	7198		2394	498
ipp035.0	1583	792	35	11497		1299	35
ipp036.0	1873	1353	134	1401		1325	115
ipp038.0	116	91	1	117		92	1
ipp039.0	110	94	0	3093		231	0
ipp040.0	384	104	1009	103		91	1028
ipp041.0	189	113	0	582		132	57
ipp042.0	86	76	9	2805		226	16
ipp043.0	131	126	2	132		126	1194
ipp044.0	94	81	2	2053		177	2
ipp045.0	109	102	2	1944		184	2
ipp046.0	102	88	13	1908		181	13
ipp047.0	96	94	0	141		96	1
ipp048.0	110	100	0	2824		224	0
ipp049.0	2144	138	36	2144		138	36
ipp050.0	94	93	0	94		93	0
ipp051.0	442	157	1	1947		231	3
ipp052.0	93	90	4	2868		236	4
ipp053.0	91	84	1	91		84	1