Changes between Version 18 and Version 19 of PS1_IPP_Czarlog_20190426

Timestamp:

May 2, 2019, 4:45:43 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -795 +795 @@
 
 
+''' Random faulted jobs ''' by JRF
+
+Some chips reporting errors in the full state:
+Do the usual attempt to clean...
+{{{
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 
+}}}
+and once cleaned, update...
+{{{
+chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 
+}}}
+
+
+
+=== Thursday : 2019.05.02 ===
+
+
+JRF: a few broken files:
+{{{
+o6601g0088o XY61    673156  1200555 update  update.LAP.PV3  LAP.PV3.20140730.20141101   LAP.ThreePi 2
+o5391g0113o XY10    192159  1202176 update  update.LAP.PV3  LAP.PV3.20140730.20141101   LAP.ThreePi 2
+o5391g0123o XY10    192168  1202408 update  update.LAP.PV3  LAP.PV3.20140730.20141102   LAP.ThreePi 2
+}}}
+
+send to clean then update:
+{{{
+  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1200555
+  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1202176
+  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1202408
+  chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1200555
+  chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1202176
+  chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1202408
+}}}
+
+
+''' Hardware fallout ''' by JRF
+
+Following a lot of nodes yo-yo-ing up and down while hardware was being sorted out this week it's worth checking what needs sorting out
+I missed putting these back up, so done:
+{{{
+neb-host --host ipp127 --state up  --note "JRF: repair -> up: fan extensions installed, look good!"
+neb-host --host ipp128 --state up  --note "JRF: repair -> up: fan extensions installed, look good!"
+neb-host --host ipp129 --state up  --note "JRF: repair -> up: fan extensions installed, look good!"
+neb-host --host ipp130 --state up  --note "JRF: repair -> up: fan extensions installed, look good!"
+}}}
+
+
+"Get three 480GB SSD's for ipp107, ipp108, and ipp117. SAMSUNG MZ7GE480HMHP or use 1TB 2.5" drives" - unclear if this was done, or what the next steps are exactly...
+I don't think this was done, so putting them back up:
+neb-host --host ipp107 --state up  --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up"
+neb-host --host ipp108 --state up  --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up"
+neb-host --host ipp117 --state up  --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up"
+ask Haydn about this.
+
+put a few more up too (ipp081 and maybe others...neb-host holds the details)
+
+Asked Haydn to keep the done items in the list in future, so that we can cross-reference afterwards.
+
+
+''' ippx005 going down suspiciously ''' by JRF
+
+To see who's been sneaking around on a machine
+{{{
+  last | less
+}}}
+We see that there has been periods of the machine rebooting (which is not great).
+
+Mentions of previous rebooting of ippx005 (and other x nodes in the same cabinet) in October's czarlog:
+{{{
+http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20181001
+}}}
+The page above mentioned that the boot time for x005 was much slower than the rest.
+
+So, let's see what ganglia says about the boot times. Indeed, multiple x nodes rebooted this morning:
+{{{ippx001-4, 5-8, 9-12, 25-28, 29-32}}}
+They all rebooted about the same time, except for '''ippx005''' which took a further hour and half to reboot..
+This is suspiciously similar to the notes from back in October, and indicates a possible problem.
+
+All of the affected nodes were in the same cabinet (13-16, 21-24, 37-40, and 41-44 are also in the cabinet but were unaffected)
+
+Had PS2 been running this could cause a real problem where we would have to bring other nodes online to handle tasks, and many jobs way get into a funny state.
+
+
+
+
+'''Feeding the update beast (how to queue more update jobs) ''' by JRF
+
+For the update.LAP.PV3 the format of the data_group is of the form:
+{{{
+  LAP.PV3.20140730.20141% = 105k --  201411% 7.4k 5/1 left, then 201412%
+  LAP.PV3.20140730.20150% = 66k -- to run, split by month 
+}}}
+So they all have the original start date of {{{20140730}}}, followed by the sub-processing date. So to queue up some more simply change the sub-date in this upcoming massive query...
+
+MEH's query for updating stuff:
+{{{
+  mysql -hscidbm -uippuser -pippuser gpc1 -B -e ' select exp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.data_group like "LAP.PV3.20140730.201412%"  group by warp_id ; ' | grep -v exp_id |  awk '{ print "mysql -hscidbm -u ippuser -pippuser gpc1 -B -e \" select s.exp_name,s.label,s.data_group,s.chip_id,s.warp_id,s.state from ( select chip_id,warp_id,exp_name,camRun.label,camRun.data_group,warpRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_id="$1" order by warp_id desc ) as s limit 1 ;\" " }' | tcsh | grep -v exp_name | grep LAP.PV3.2014 | grep cleaned | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$4" ; chiptool -dbname gpc1 -setimfiletoupdate  -set_label update.LAP.PV3 -chip_id  "$4" ;warptool -dbname gpc1  -updaterun -set_label update.LAP.PV3 -warp_id "$5"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$5 }' | head -n 1000 | tcsh
+}}}
+The gist of this is to:
+ - get exp_ids related to the data_group specified
+ - get the latest state (given by whichever warp is latest for that exposure)
+ - if the state is cleaned then queue it for update!
+ - and the updates are limited to 1000 at a time.
+
+To see how far along we are you use this query. As of today we have ~260k warps done (nice).
+{{{
+  select count(*),state,label,data_group from chipRun where label like "update%" group by label,state;  select count(*),state,label,data_group from warpRun where label like "update%" group by label,state ;
+}}}
+
+So, the next one for us to do is '''201412''', so change that in the big-ass query and send it. I've run it twice to set 2000 or so jobs to go. I'll queue some more tomorrow too
+
+
+''' Added ippc76 as neb apache server ''' by JRF
+
+modified the {{{nebservers.csh}}} file to include it. We had check the setup and functionality earlier this week.
+
+
+
 
 TODO:
- - read up on the desperate diffs scripts so far
- - add ippc76 as an apache node (it has been performing well for mops stdscience)
-
-
-=== Thursday : 2019.05.02 ===
+ - TONIGHT (provided we get data): read up on the desperate diffs scripts so far - Run some diff checkings while data it processing 
+ - fix those 0 size psf files
+ - A stamp request was made form MPIA, do we want to be supporting this? Will be raised at tomorrows meeting by MEH.
+ - Firday: queue more jobs for the next czar Roger
+