Changes between Version 16 and Version 17 of PS1_IPP_Czarlog_20190318
- Timestamp:
- Mar 25, 2019, 3:55:05 PM (7 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
PS1_IPP_Czarlog_20190318
v16 v17 4 4 (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) 5 5 6 Czar of the week: John, with MEH as czar-lord (previously: TdB )6 Czar of the week: John, with MEH as czar-lord (previously: TdB + EAM) 7 7 8 8 === Monday : 2019.03.18 === … … 53 53 * MEH: ippb04, b05 causing many i/o faults from the rsyncs -- set neb-host down from repair since many primary copies of files there... -- will need to temporarily set back to repair to clear other faults for the cases where files are only on those machines 54 54 55 JRF: (afternoon czar formation ft. MEH) 56 55 ''' Afternoon czar formation (ft. MEH)''' by JRF 57 56 58 57 The recipes directory contains a bunch of configurations: … … 91 90 chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0669o 92 91 }}} 93 As you can see we're using the NOMASKTEST reduction that we just had a look at the config for.92 As you can see we're using the NOMASKTEST reduction (which we just had a look at). 94 93 95 94 Don't forget to add the label to pantasks to make it run! 96 95 96 ''' stuck jobs ''' by JRF 97 97 98 98 Time to fix some stuck jobs! … … 114 114 remove '-simple' to get some more information. 115 115 116 JRF (evening): Jobs approaching 100k, so restarting stdscience pantasks. A bunch of failed jobs are beginning to accumulate too, but I'll save that fun for the morning.116 '''Later that same evening''': Jobs approaching 100k, so restarting stdscience pantasks. A bunch of failed jobs are beginning to accumulate too, but I'll save that fun for the morning. 117 117 118 118 … … 123 123 124 124 125 JRF (afternoon) - czar formation ft. MEH: 126 127 128 Maybe tomorrow ecliptic.RP 129 125 ''' czar formation (ft MEH)''' by JRF 126 127 ''' warp fault stuck in full state ''' by JRF 130 128 131 129 Weird warp fault where the state is full: … … 151 149 152 150 153 '''Solving warp issues''' 151 '''Solving other warp issues''' 152 154 153 Move the stuff that is currently broken under the label 'update.OSS' to the 'update.OSS.broke' label. 155 154 To figure out which ones need to be moved have a look at the state, fault, and skyfile … … 184 183 | xargs -0 bash -c 185 184 186 187 }}} 188 Note: The above its formatted for readability, you'll have to squish it into one line, fill in the XXXXXs, and remove pretend. 185 }}} 186 '''Note''': The above its formatted for readability, you'll have to squish it into one line, fill in the XXXXXs, and remove pretend. 189 187 190 188 … … 200 198 MEH moved across the remaining update.OSS.broke labels. 201 199 202 The SVN server died! 203 204 May want to figure out which exposures might have already been reprocessed too, as these ones do not need to be done. 205 we are going to add the LAP PV3 label 206 207 208 There are some broken stamps that Rob has requested. 209 210 {{{ 211 count label req_id name timestamp fault exp_id dtime utc_timestamp() 200 The SVN server died! EAM has been moving a backup of it to a new server. 201 202 203 ''' Broken/pending MOPS stamp requests''' 204 205 There are some broken stamps that Rob has requested: 206 {{{ 207 count label req_id name timestamp fault exp_id dtime utc_timestamp() 212 208 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466358 00:04:05 2019-03-20 22:20:01 213 209 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466379 00:04:05 2019-03-20 22:20:01 … … 256 252 }}} 257 253 258 259 Going back to the stuff that had hung pstamps, on the pstamp server we can see that nothing has moved for about 10 hours. 260 261 262 If you come across a stamp that you suspect maybe be stuck, have a look at it using: 263 {{{ 264 pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 265 }}} 266 remove '-simple' to get some more information. 267 268 Going back to checking out Rob's stuff: 269 {{{ 270 SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068); 271 +-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+ 272 | exp_name | exp_id | chip_id | label | state | cam_id | state | warp_id | label | state | 273 +-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+ 274 | o6002g0321o | 466358 | 1067421 | ps_ud_MOPS | full | 1036073 | full | 1018412 | ps_ud_MOPS | update | 275 | o6002g0340o | 466379 | 1067424 | ps_ud_MOPS | full | 1036086 | full | 1018431 | ps_ud_MOPS | update | 276 | o6698g0496o | 705045 | 1078654 | ps_ud_MOPS | full | 1040692 | full | 1022614 | ps_ud_MOPS | update | 277 | o6698g0517o | 705068 | 1078658 | ps_ud_MOPS | full | 1040696 | full | 1022879 | ps_ud_MOPS | update | 278 +-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+ 279 }}} 280 281 There are a still a bunch that have warps in the update state. 254 This did not fully succeed, they remain stuck in the update state.. 282 255 I'm going to try cleaning... 283 256 {{{ … … 294 267 chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1078658 -set_label ps_ud_MOPS 295 268 }}} 296 297 '''More pstamp investigations:''' 269 This sometimes fixes the problem. 270 271 272 ''' Hung stamp requests ''' by JRF 273 274 Going back to the stuff that had hung pstamps, on the pstamp server we can see that nothing has moved for about 10 hours. 275 If you come across a stamp that you suspect maybe be stuck, have a look at it using: 276 {{{ 277 pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 278 }}} 279 remove '-simple' to get some more information. 280 281 282 '''More pstamp investigations:''' by JRF 298 283 299 284 {{{ … … 309 294 | exp_name | exp_id | chip_id | label | state | cam_id | state | warp_id | label | state | 310 295 +-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+ 311 | o5591g0062o | 289960 | 830503 | ps_ud_MOPS | update | 806578 | full | 786578 | PI_20110130.reprocessing.20130426 | drop |312 296 | o5591g0062o | 289960 | 830503 | ps_ud_MOPS | update | 806578 | full | 788140 | ps_ud_MOPS | update | 313 | o5591g0083o | 289981 | 830524 | ps_ud_MOPS | update | 806604 | full | 786618 | PI_20110130.reprocessing.20130426 | drop |314 297 | o5591g0083o | 289981 | 830524 | ps_ud_MOPS | update | 806604 | full | 788180 | ps_ud_MOPS | update | 315 298 +-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+ … … 336 319 337 320 338 '''Without a mask''' 321 '''Without a mask''' by JRF 339 322 340 323 Rob has requested some things to be processed without a mask: … … 344 327 o7255g0519o 04:22:26.323 +21:06:18.76 K17W15C 345 328 346 So I've queued them up 329 So I've queued them up to reprocess without a mask i.e. {{{-set_reduction NOMASKTEST}}} 347 330 {{{ 348 331 chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0537o … … 353 336 They go through fine 354 337 355 '''Broken IPP jobs''' 338 339 '''Broken IPP jobs: label changing + more''' by JRF 356 340 357 341 There are two warp_ids that have problems: '''1640466, 1640460''' … … 369 353 }}} 370 354 371 For exercises sake, we're going to change the label for the first line's chip and warp id to ps_ud_MOPS :355 For exercises sake, we're going to change the label for the first line's chip and warp id to ps_ud_MOPS (as it's related to some stamps that MOPS are after): 372 356 {{{ 373 357 chiptool -dbname gpc1 -updaterun -chip_id 1693054 -state update -set_label ps_ud_MOPS … … 375 359 }}} 376 360 377 Alternatively, you could pipe the results from pstamptool to do the update like so: 378 {{{ 379 pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh 380 381 pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh | awk '{ print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id "$23" ; warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$1 }' 382 }}} 383 384 '''More IPP monitor errors (solving missing log)''' 361 362 '''More IPP monitor errors (solving missing log)''' by JRF 385 363 386 364 Checking out some of the errors, one of them complains about a file that it could not use: … … 396 374 }}} 397 375 398 This shows that the other two copies of this file are not empty (see the checksum: d41d8cd98f00b204e9800998ecf8427e)376 This shows that the other two copies of this file are not empty, which is good (see the checksum: '''d41d8cd98f00b204e9800998ecf8427e''') 399 377 {{{ 400 378 36.pswarp.mdc … … 420 398 1) delete the empty file 421 399 2) replace the empty file with one of the correct ones. 422 We 're chose option 1.400 We chose option 1. Simply copy one of the good ones over the old one 423 401 424 402 … … 487 465 chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1257054 -set_label update.OSS.broke 488 466 489 And so the chip state updated to clean...but the warp tool is still in update .490 491 Most things just seem stuck on the old missing files from ippb05 problem (so I put it into repair for the time being).467 And so the chip state updated to clean...but the warp tool is still in update (later in here). 468 469 Most other things just seem stuck on the old missing files from ippb05 problem (so I put it into repair for the time being). 492 470 {{{ 493 471 chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1268153 -set_label update.OSS.broke … … 504 482 === Thursday : 2019.03.21 === 505 483 506 '''There are a couple of broken stamp requests''' 507 484 485 '''Broken stamps -> assertion failures -> reprocessing''' by JRF 508 486 509 487 Tasks that are pending, as found by: … … 517 495 10959535 new warp 475957 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959535.log 500 518 496 }}} 497 519 498 Let's query these: 520 499 {{{ … … 529 508 +---------+----------------------------+--------+---------+----------------------------+--------+ 530 509 }}} 531 These one are in update, and under the broke label. 510 511 These ones are in update, and under the broke label. 532 512 So we'll change their labels to ps_ud_MOPS so that they get processed and can diagnose the errors from there: 533 513 {{{ … … 598 578 599 579 600 601 '''We need to add a stamp for the next set of tasks after update.OSS''' 580 '''Adding a new label (that persists on restart)''' by JRF 602 581 603 582 The LAP.PV3 stuff needs to updated next, and this will take a while. So it makes sense to add a new label that will be loaded automatically when pantasks starts up. 604 583 605 To do this simply add the following li ke to the 'input' file in the stdscience directory:584 To do this simply add the following line to the 'input' file in the stdscience directory: 606 585 {{ 607 586 add.label update.LAP.PV3 … … 625 604 626 605 627 '''There are some broken jobs on IPP monitor''' 628 629 Most had memory problems and needed reverting.606 '''There are some broken jobs on IPP monitor''' by JRF 607 608 Most had memory problems and only needed reverting. 630 609 631 610 632 611 === Friday : 2019.03.22 === 633 612 634 '''Broken jobs breakfast '''613 '''Broken jobs breakfast saga''' by JRF 635 614 636 615 What better way to start the day than a cup of coffee and some red stalled jobs on IPP monitor. 616 617 '''ps_ud_MOPS chip faults''' by JRF 637 618 638 619 A ps_ud_MOPS job stuck in the chip stage: … … 683 664 {{{ 684 665 chiptool -dbname gpc1 -updaterun -chip_id 1280264 -set_state goto_cleaned -set_label goto_cleaned 685 warptool -dbname gpc1 -updaterun -warp_id ??????? -set_state goto_cleaned -set_label goto_cleaned 686 }}} 687 688 Exp Name Class Exp ID Chip ID State Label data grp dist grp Fault 689 o5426g0329o XY26 208836 1345211 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20141219 LAP.ThreePi 2 690 Trying to revert this failed...so reprocess it (it's i-band): 691 {{{ 692 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o5426g0329o 693 }}} 694 Reprocessed ok, time to cleanup the old stuff: 695 {{{ 696 chiptool -dbname gpc1 -updaterun -chip_id 1345211 -set_state goto_cleaned -set_label goto_cleaned 697 warptool -dbname gpc1 -updaterun -warp_id 1301704 -set_state goto_cleaned -set_label goto_cleaned 698 }}} 699 700 701 '''warp''' 666 warptool -dbname gpc1 -updaterun -warp_id 1260815 -set_state goto_cleaned -set_label goto_cleaned 667 }}} 668 669 Repeat for this process for the other failed chip job 670 671 672 '''ps_ud_MOPS warp faults''' by JRF 702 673 703 674 There are two ps_ud_MOPS stuck in the warp stage: … … 734 705 735 706 736 737 707 '''Lots of update.LAP.PV3 have missing logs''' 738 708 … … 745 715 This looks different to yesterday's issue... they are 746 716 747 717 {{{ 748 718 SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state,warpRun.label warp_label,warpRun.data_group,warpSkyfile.fault,warpSkyfile.path_base FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.warp_id = '1302416' AND warpSkyfile.skycell_id like 'skycell.0788.077' LIMIT 80; 749 750 Maybe do this from earlier in the week: 719 }}} 720 721 Maybe do this from earlier in the week: 751 722 Weird warp fault where the state is full: 752 723 {{{ … … 778 749 }}} 779 750 751 780 752 That solved the majority, 139/141, issues. Two issues remain: 781 753 1) is a fault 4 so just revert it 782 2) the other is a fault 3, but has the same issues as the above, where the state is full but it is failing to find a specific log file. So -> set to clean -> set to update ?754 2) the other is a fault 3, but has the same issues as the above, where the state is full but it is failing to find a specific log file. So -> set to clean -> set to update 783 755 {{{ 784 756 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1079367 … … 789 761 790 762 791 '''Broken chip in update.OSS''' 792 793 '''Broken warp in update.OSS''' 763 '''Broken warp in update.OSS''' by JRF 794 764 795 765 The broken warp in update.OSS wants ippb04...so will need to put that up into repair and revert the warptool job. 796 766 797 767 798 799 800 801 '''Maybe Rob job not all good''' 802 803 Have a look at theses exposures: 768 '''Maybe Rob job not all good''' by JRF 769 770 Have a look at ippitc email about pending stamps: 804 771 {{{ 805 772 1 MOPS 918892 weryk_5C95386D 2019-03-22 20:19:52 25 217487 00:10:09 2019-03-22 20:30:01 … … 823 790 The warps look good..., so maybe no problem here 824 791 825 Still a bunch of pending things :792 Still a bunch of pending things according to this: 826 793 {{{ 827 794 pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple … … 832 799 }}} 833 800 834 Use the above to update theruns, aka:835 {{{ 836 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1247225837 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1196545801 The later updates both runs, aka: 802 {{{ 803 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id xxxxxx 804 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id xxxxxx 838 805 }}} 839 806 … … 846 813 847 814 848 ''' ganglia demon failed across the cluster ''' 849 850 JRF: Had to restart it on a whole load of machines... 815 ''' ganglia demon failed across the cluster ''' by JRF 816 817 JRF: Had to restart it on a whole load of machines... MEH suggested: 851 818 {{{ 852 819 sudo /etc/init.d/gmond status … … 860 827 Though the above didn't always work on the ubuntu machines 861 828 862 ''ps_ud_MOPS error''' 863 864 o6473g0463o XY44 627000 1347469 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20141220 LAP.ThreePi 3 PS1GPC1829 ''ps_ud_MOPS error''' by JRF 830 831 o6473g0463o XY44 627000 1347469 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20141220 LAP.ThreePi 3 PS1 GPC1 865 832 Tried to revert, it failed. Then tried to update, it also failed. 866 833 … … 876 843 877 844 878 ''' pstamp s '''845 ''' pstamp problems ''' by JRF 879 846 880 847 {{{ … … 912 879 913 880 === Sunday : 2019.03.24 === 881 882 883 ''' chips stuck in full ''' 914 884 915 885 There are a bunch of chips and warps stuck in the full state... … … 956 926 }}} 957 927 HOWEVER, this failed and I'm not sure why... 928 958 929 So there are now some fault 2 entries with goto_cleaned... query this to get them: 959 930 {{{ … … 961 932 }}} 962 933 963 FIX ON MONDAY!964 965 '''Remaining chip issues:''' 966 967 All 4 of them: 968 o5280g03 01o XY01 151113 1059611 updateupdate.LAP.PV3969 o5280g031 7o XY01 151129 1059616 updateupdate.LAP.PV3970 o 5280g0316o XY01 151128 1059817 updateupdate.LAP.PV3971 o6091g0094o XY22 498248 1060186 update update.LAP.PV3 972 973 Complain about a missing mdc config file:934 DEFERED UNTIL NEXT WEEK! 935 936 '''Remaining chip issues:''' by JRF 937 {{{ 938 o5280g0301o XY01 151113 1059611 update update.LAP.PV3 939 o5280g0317o XY01 151129 1059616 update update.LAP.PV3 940 o5280g0316o XY01 151128 1059817 update update.LAP.PV3 941 o6091g0094o XY22 498248 1060186 update update.LAP.PV3 942 }}} 943 944 They all complain about a missing mdc config file: 974 945 {{{ 975 946 WARNING: Config dump file neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc is missing. Using current recipes and file rules.
