PS1 IPP Czar Logs for the week 2019.03.18 - 2019.03.24

(Up to PS1 IPP Czar Logs)

Czar of the week: John, with MEH as czar-lord (previously: TdB)

Monday : 2019.03.18

• TdB: In the morning (05:45), I noticed ippb18 was reported as down by nagios. When trying to get more information, I noticed neb-host was reporting too many connections and no machines where showing up in IPP monitor (all greyed out). Following a quick back and forth with Gene I then set all pantasks to stop using:

  check_system.sh stop
  

  Furthermore, the compute nodes ippc71-75 were showing huge load on ganglia, so we turned off apache on those machines, using:

  /etc/init.d/apache2 stop
  

  as root on each of the machines.

• Following this, we (John, Roger, Gene, myself) all came in at 09:00 to diagnose and fix the issue. First, we looked at the dump of processlist on ippc75:

  mysql -h localhost -u root -e "show processlist" > neb.proc.20190318
  

  This showed 64 connections from ippc76, which is not on the list of nebservers (see ~ippitc/ippconfig/nebservers.csh) and also showed soem connections from ippc28. Looking at the configuration file /etc/apache2/modules.d/apache2-mod_perl-startup.pl showed 256 MaxClients connections set up for the compute nodes. Machine ippc76 has StartServers = 64, so it will start with normally 64 connections, but could go up to 256 total. Combining all machines, this results in a max of 1349 connections, which is a lot.

Looking at the max connections in MYSQL using:

select @@max_connections;

shows that the maximum is set to 1024, which is less than 1349, so that could be a reason for the failure. We are setting that higher now, to avoid this issue:

SET GLOBAL max_connections = 2048 (in mysql)

The full reason for the break is not totally clear, but the huge load spikes on the apache servers roughly coincide with ippb18 going down, so that might well be related.

For now, we are setting the apache servers back to on, using:

/etc/init.d/apache2 start

as root on each of the machines.

Finally, the pantasks, are set back to run using:

check_system.sh run

• MEH: bulk updates now continuing again in ~ippmops/stdscience
• MEH: sending more chips to cleanup for warps in full state
  • ipp091 excess load from being harassed from cleanup and failed BBU, leaving in neb-host repair now
• MEH: ippb04, b05 causing many i/o faults from the rsyncs -- set neb-host down from repair since many primary copies of files there... -- will need to temporarily set back to repair to clear other faults for the cases where files are only on those machines

JRF: (afternoon czar formation ft. MEH)

The recipes directory contains a bunch of configurations:

ippitc/src/ipp-20170121/ippconfig/recipes

We'll look at the reductionClasses.mdc config file:

Contains the configuration for how to process something. In this case the CHIP_PPIMAGE stage is set to CHIP_NOMASKTEST instead of the usual CHIP.

# test for MOPS request reprocessing without masking

NOMASKTEST             METADATA
        CHIP_PPIMAGE      STR     CHIP_NOMASKTEST
        .
        .
        .
        BACKGROUND_PSWARP       STR     BACKGROUND
        PSVIDEOPHOT             STR     PSVIDEOPHOT
END

There are separate configs for each camera (go up a directory then into the relevant camera dir). There are a bunch of configs here are empty in the main config, but have entries for the separate cameras. This is because data from each camera should be treated separately.

WATCH OUT: If you reprocess something with a different config then the stamp server will use whatever the latest config was for a request! Therefore, it's best to do some crazy reprocessing locally, so as not to pollute the database with entries that could confuse the current stamp request system.

Time to reprocess some stuff for Rob with no masks (as seen on PSNSC_MOPS page and czar log from 2019.02.18 - 2019.02.24):

chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0726o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0707o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0688o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0669o

As you can see we're using the NOMASKTEST reduction that we just had a look at the config for.

Don't forget to add the label to pantasks to make it run!

Time to fix some stuck jobs! Some files are needed from ippb05, but it has been misbehaving and was set to down ealier today. We shall quickly put it to repair so that the files can be accessed:

neb-host ippb05 repair --note "JRF: down -> repair: so that some broken updates can clear on ippmonitor"

Then we shall revert the chip for the stuck job so that it can continue:

chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1

You don't need to specify the chip_id, it'll just revert all faults that are stuck.

If you come across a stamp that you suspect maybe be stuck, have a look at it using:

pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple

remove '-simple' to get some more information.

Tuesday : 2019.03.19

Wednesday : 2019.03.20

Thursday : 2019.03.21

Friday : 2019.03.22

Saturday : 2019.03.23

Sunday : 2019.03.24