Changes between Version 28 and Version 29 of PS1_IPP_Czarlog_20120716

Timestamp:

Jul 19, 2012, 2:21:46 PM (14 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20120716

@@ -27 +27 @@
  * 11:00 adding back 1x compute2+1x stare+1x wave4 for 103 nodes, stsci node spikes are broader now (time) -- monitoring -- seems okay and reaching >400/hr
  * 11:30 ganglia reports ipp008 all memory swapped but isnt. gmond restart worked for a little bit but back to saying all memory swapped. keep eye on.
- * 12:00 ecliptic.rp chip faults often NON-EXISTANT, but are being found with the new deneb-locate.py command -- details later
+ * 12:00 ecliptic.rp chip faults often NON-EXISTANT, but are being found with the new deneb-locate.py command -- 
+{{{
+        o5362g0549o     XY36    181808  488192 
+neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits
+      0                     NON-EXISTANT file:///data/ippb02.1/nebulous/d1/13/917781576.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+      0                     NON-EXISTANT file:///data/ippb02.0/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+
+-- rogue files?
+/home/panstarrs/ipp/local/bin/deneb-locate.py gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits
+/data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits -- seems ok
+/data/ipp024.0/nebulous/d1/13/319361864.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits -- 0 size
+
+-- copy over to non-existant file location
+cp /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits /data/ippb02.1/nebulous/d1/13/917781576.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+cp /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits /data/ippb02.0/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+
+-- want to replicate to main machines so first instance is not looked for on ippb machines
+--> could also replicate all off of ippb machines since overly full -- if not use volume then likely puts to stsci randomly and don't want
+neb-replicate neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ipp030.0
+neb-replicate neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ipp031.0
+
+neb-cull neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ippb02.0
+neb-cull neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ippb02.1
+
+-- once reverted and okay, can rm rogue files
+rm /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+rm /data/ipp024.0/nebulous/d1/13/319361864.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
+
+}}}
  * 12:30 adding back 1x compute+1x compute2 for 124 nodes in stack pantasks -- gets the stsci nodes to load spike to >100 with nfs.. but handling so far..
- * 12:50 pile up of ecliptic.rp in camera stage, ~15 runs >20ks. killing and reverting to see if clear ok.
+ * 12:50 pile up of ecliptic.rp in camera stage, ~15 runs >20ks. killing and reverting to see if clear ok -- when reverted still stall -- need to be looked into.
  * 13:58 CZW: Added SAS_v8 to stdscience.  Defined LAP queue for Kepler field (based on http://kepler.nasa.gov/Science/about/targetFieldOfView/, defined as (227.5,36)->(302.5,52.5)) and set it as the "current.queue".  This means that as the 09:00hr slice finishes, it will launch Kepler fields.  I've also defined a 10:00hr slice, so we can switch to that when Kepler is fully queued.
  * 16:00 short testing single stage processing for chip/camera/warp/stack, warp is the primary culprit for harassing the stsci machines.