IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20190510

Version 16 (modified by mhuber, 7 years ago) ( diff )

--

PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16

    1. PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16
      1. Friday : 2019.05.10
      2. Saturday : 2019.05.11
      3. Sunday : 2019.05.12
      4. Monday : 2019.05.13
      5. Tuesday : 2019.05.14
      6. Wednesday : 2019.05.15
      7. Thursday : 2019.05.16

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.10

CZAR HANDOVER by CCL

  • Haydn mentioned he will check them on 5/13 (remind him maybe)
  • ipp128: Device: /dev/bus/0 [megaraid_disk_15], Read SMART Self-Test Log Failed
  • ipp139: Device: /dev/bus/0 [megaraid_disk_37], Read SMART Self-Test Log Failed
  • ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
  • ippc129, ippc135, ippc145, ippc149, ippc150: drive issue
  • update.LAP.PV3: 15k warp prcessing jobs on ippmonitor, some of them might have been processed to be "full" state by queue their data_state in the warpSkyfile table. Their data quality is not equal to 0 which may affect the state keep "update".
  • QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
  • these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh
  • TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues: 
    mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcsh
    I took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.
  • TdB: Around 18:00 we hit a situation where chips were still updating but warps were not. I did a bit of digging: 
    mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "update" group by chip_id limit 500; '

There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through: 

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "full" and warpRun.label LIKE "update.LAP.PV3.lazy" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

Saturday : 2019.05.11

  • TdB: One exposure of the night was ruined, which caused chunk OSSR.R13S1.12.Q.i to be incomplete. Check the status and see if a desperate diff needs to be made: 
    mysql -hscidbm -uX -pX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;"
    which returns: 
    chunk	object	warp1	warp2	warp3	warp4	diff1	diff2	diff3	diff4	workdir	label	data_group	reduction
OSSR.R13S1.12.Q.i	ps1_28_0298	2107392	2107411	2107431	2107450	1764479	1764479	1764537	1764537	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0299	2107384	2107405	2107423	2107444	1764467	1764467	1764525	1764525	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0303	2107399	2107418	2107438	2107456	1764488	1764488	1764546	1764546	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0304	2107390	2107408	2107428	2107449	1764473	1764473	1764536	1764536	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0305	2107388	2107407	2107426	2107446	1764472	1764472	1764528	1764528	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0310	2107397	2107416	2107436	2107454	1764487	1764487	1764543	1764543	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0391	2107383	2107404	2107425	2107443	1764468	1764468	1764519	1764519	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0399	2107400	2107421	2107441	2107459	1764497	1764497	1764556	1764556	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0400	2107389	2107414	2107430	2107451	1764485	1764485	1764538	1764538	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0401	2107391	2107410	2107429	2107448	1764478	1764478	1764531	1764531	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0402	2107382	2107403	0	2107442	1764462	1765059	0	1765059	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0403	2107385	2107402	2107420	2107440	1764461	1764461	1764520	1764520	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0410	2107398	2107419	2107437	2107458	1764495	1764495	1764554	1764554	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0412	2107395	2107412	2107434	2107453	1764480	1764480	1764544	1764544	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0413	2107393	2107415	2107432	2107452	1764486	1764486	1764539	1764539	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0414	2107387	2107409	2107424	2107445	1764474	1764474	1764527	1764527	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0415	2107386	2107406	2107427	2107447	1764469	1764469	1764532	1764532	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0416	2107381	2107401	2107422	2107439	1764457	1764457	1764516	1764516	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0426	2107396	2107417	2107435	2107457	1764493	1764493	1764547	1764547	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0428	2107394	2107413	2107433	2107455	1764484	1764484	1764545	1764545	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
    In fact, it has already been diffed, since we see two separate diff_ids for this obejct, diff_id=1764462 (made from 2107382 and 2107403) and diff_id=1765059 (made from 2107403 and 2107442). I am guessing the nightly_science.pl script kicked in here, at the end of the night.

----------------

There are some warps complaining about missing .mk files. First set them to update to see if that fixes the problem: 

warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292074
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292075
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292079

failing that, send to reprocess: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0268o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0280o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6315g0144o

and clean up their chip and warp ID: 

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342029 
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342030 
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342034 

warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292074
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292075
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292079

--------------------

There are also some I/O relatd errors with ippb18, with fault codes 3 and 4. Judging by Ganglia, it is seeing some fairly heavy usage. Send the exps to revert, but put ippb18 into repair for the time being to give it a break. 

neb-host --state repair --host ippb18 --note 'TdB: up->repair, due to it getting hammered and producing I/O errors'

--------------------- fixing some errors. send to clean and back 

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1407389

chiptool -dbname gpc1 -updaterun -set_label ps_ud_WEB -chip_id 1407389
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1407389

--------------------- night: ippc112 is not behaving well. It appears to be down, and repeated power cycling is not helping. Mark removed it from processing and I notified Haydn about it.

Sunday : 2019.05.12

  • TdB: Machine ippb18 is now down on ganglia (though no email from nagios) and the console display log is listing EDAC errors. I put it to neb-host down and powered it off in the console (it was already reported as down anyways). Then, I emailed Haydn about it.

Fixing some missing .mk files again: 

warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1295348

Didn't work. send to reprocess: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5231g0024o

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1344272
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1295348

Also some corrupted memory blocks that need reprocessing: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6235g0387o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6284g0320o

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1364687
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1413646
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1334220
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1363203

*TdB: 18:00 we have finished going through the chip exposure that Roger queued up, but there are still ~6k warps that are lagging behind and are imbalanced. In principle it would be ok to run with just warps updating to remove the imbalance, but might as well feed in a small batch of chip+warps anyway, since warp run at a higher rate than chips. But unsure which data_group we were doing, since I could find no info on the czarlog from last week about which daa_groups Roger queued up. Looking at the query John provided we should be at data_group = LAP.PV3.20140730.20150% (but please correct me if I am wrong). Therefore, setting a batch of 1000 exp to update, using: 

mysql -hscidbm -uX -pX gpc1 -B -e ' select exp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.data_group like "LAP.PV3.20140730.20150%"  group by warp_id ; ' | grep -v exp_id |  awk '{ print "mysql -hscidbm -u ippuser -pippuser gpc1 -B -e \" select s.exp_name,s.label,s.data_group,s.chip_id,s.warp_id,s.state from ( select chip_id,warp_id,exp_name,camRun.label,camRun.data_group,warpRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_id="$1" order by warp_id desc ) as s limit 1 ;\" " }' | tcsh | grep -v exp_name | grep LAP.PV3.2014 | grep cleaned | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$4" ; chiptool -dbname gpc1 -setimfiletoupdate  -set_label update.LAP.PV3 -chip_id  "$4" ;warptool -dbname gpc1  -updaterun -set_label update.LAP.PV3 -warp_id "$5"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$5 }' | head -n 1000 | tcsh

Monday : 2019.05.13

*TdB: Check on the disk space of some of the machines, following cleanup:

Machines ipp123-126 have filled up their disks. Putting them to repair to prevent more data going there. Also ipp110.0 is full 

neb-host --state repair --host ipp123 --note 'TdB: up->repair, disk is getting very full'
neb-host --state repair --host ipp124 --note 'TdB: up->repair, disk is getting very full'
neb-host --state repair --host ipp125 --note 'TdB: up->repair, disk is getting very full'
neb-host --state repair --host ipp126 --note 'TdB: up->repair, disk is getting very full'
neb-host --state repair --host ipp110 --note 'TdB: up->repair, disk is getting very full'

But the recent cleanup has freed enoguh space on ipp106 and ipp107 to allow them to go back to up. 

neb-host --state up --host ipp106 --note 'TdB: repair->up, disk space freed in cleanup'
neb-host --state up --host ipp107 --note 'TdB: repair->up, disk space freed in cleanup'
  • MEH: cleanup chip updates example 
    mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }'

mysql -hippdb08 -uX -pX gpc1 -B -e ' select count(chip_id) from chipRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(warp_id) from warpRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(diff_id) from diffRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(stack_id) from stackRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(dist_id) from distRun where state like "goto_cleaned" and label like "goto_cleaned"; '; date
  • TdB: afternoon: Some missing .mk files in the warps. Set them to update the skyfile 
    warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1342050
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1344264
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1358970
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1365663
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1368220
    Didn't work. send to reprocess from scratch, instead.

Also some memory blocks that need reverting: 

warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1

-----------------------

Some of the warp update processing got stuck, probably due to their chip exps being in wait state. Query those and send them to cleanup, followed by an update. Pump them to a file first 

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "wait" group by chip_id; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1"; chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$1"; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$1 }' > LAP.PV3_waits

There are 1389 of them. Send them to cleanup and then back to update. That solved most of the problems with the remaining update warps

----------------------

IPP meeting: Cleanup:

We should periodically clean up the chip stage pixels for the updates we are running, to preserve disk space. Use the following query, and queue up a 1000 or so once in a while, or a few thousand overnight: 

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcsh

----------------------

Haydn wants to swap some drives from ippc17 and ippc90. Use check_system.sh to remove them from processing: 

./check_system.sh hostoff ippc17 (do this a bunch of times)
./check_system.sh hostoff ippc90

After the swap, put them back up 

./check_system.sh hoston ippc17 (do this a bunch of times)
./check_system.sh hoston ippc90

----------------------

Chips with wait states: There is still the ~700k chip exps with a state=wait that need to be dealt with. We can query the labels and data_groups that they occupy, using: 

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY data_group,label) as s ORDER by waitct ASC;'

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY label) as s ORDER by waitct ASC;'

The vast majority of them are in fact exps which have a label of goto_clean% which are likely previously cleaned chip exps. However, we cannot simply set their state to cleaned, since we have no way of telling whether the cleanup operation went succesfully or failed. Therefore, we need to send the whole lot back to cleanup. Nonetheless, by parsin the xps, we can preferentially send exps that we still need to use in the update processing before doing things previosuly cleaned. Focus on labels like %LAP.PV3% or %OSS% at first and slowly end them to cleanup in between other cleanup jobs.

  • TdB: at around 17:30, the IPPmonitor stalled. Turns out, mysql crashed on ippdb05. Let's check things:

Look at the error log under /var/log/mysql (log into ippdb05 as own user) 

190513 17:16:46 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=262144
max_used_connections=15
max_connections=2048
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1605632 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x7f664c587e30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x7f664c587e30, backtrace may not be correct.
Bogus stack limit or frame pointer, fp=0x7f664c587e30, stack_bottom=0x44c90000, thread_stack=262144, aborting backtrace.
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=4
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
terminate called without an active exception

So, let's try to restart mysql on ippdb05 and see if ti comes back up. As root, do: 

sudo /etc/init.d/mysql start
 * Starting mysql ...
 * Strange, the socket file already exist in "/var/run/mysqld/mysqld.sock"
 * it will be removed now and re-created by the MySQL server
 * BUT please make your checks.
 * Starting mysql (/etc/mysql/my.cnf)
 * MySQL NOT started (0)

But, it probably did start. Check, using: 

ps aux | grep mysql
mysql    27881 93.3  7.5 146558796 14950952 ?  Ssl  18:25   0:33 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock

Indeed, it started just fine.

Checking the error log file again shows: 

190513 18:26:29  InnoDB: Started; log sequence number 5173 4202608365
190513 18:26:29 [Note] Recovering after a crash using /var/lib/mysql/mysqld-bin
190513 18:26:29 [Note] Starting crash recovery...
190513 18:26:29 [Note] Crash recovery finished.
190513 18:26:29 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.

InnoDB: Rolling back of trx id 2 1776754758 completed
190513 18:26:29  InnoDB: Rollback of non-prepared transactions completed
190513 18:26:29 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.77-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo Linux mysql-community-5.0.77-r1
  • MEH: starting some older diff cleanup now to help free up space on lower data nodes

Tuesday : 2019.05.14

Wednesday : 2019.05.15

Thursday : 2019.05.16

Note: See TracWiki for help on using the wiki.