PS1 IPP Czar Logs for the week 2019.05.03 - 2019.05.09

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.03

pstamp requests by JRF

Received some error emails about pstamp requests. checking out the pstamp server we can see that over 800,000 jobs have been submitted by QUB.

The errors come from a script run by the crontab on ippc26 (this is mentioned in the emails). The script checks if pstamp has over 100,000 jobs done as this is were pantasks becomes flaky.

Checking out the status of the pstamp pantasks keep an eye on:

  request.finish.run

as we DO NOT want to restart if this is running as it will cause funkiness.

So let's restart pstamp, doing the usual:

  pantasks> stop
  pantasks> shutdown now    --check that the jobs have finished first

  ~ippitc/pstamp> pantasks_server & pantasks_client
  pantasks> server input input
  pantasks> setup
  pantasks> run

I have added ippc76 to the hosts_apache group in the ~ippitc/ippconfig/pantasks_hosts.input file. This is so that when the nodes are allocated that ippc76 will be turned off when the apache group is removed.

Since over 800,000 requests were made the pstamp pantasks will need restarting about every hour or so.

• JRF: ippc72 has died, sent an email to Haydn, left the PDU off. Also commented it out of the ~ippitc/ippconfig/nebservers.csh file.

CZAR HANDOVER by JRF

• ipp139 has started having issues so there will likely be some liaising with hardware people to fix that.
• ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
• 'wait' state: the past week has seen some chip states set to 'wait' which is not a valid state. They can appear in IPP processing at times, and the quickest way to fix them is: change label + imfile update:

    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id XXXXXX
    chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id XXXXXX
  

  the update second part is needed as the change from wait to update isn't usually recognized.
• More updates will need to be queued, details of how to do this are given in the Thursday 2019-05-02 czarlog
• Guide for desperate diffs is in the czarlog for last week. HOWEVER, TdB likely has a better way of doing it by now
• QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).

CCL took over after 2pm

• CCL: need to modify /home/panstarrs/ipp/local/bin/apachedisk_chk.sh as ipp on ippc19 to solve the crontab error message, i.e., remove 72 & add 76 for host list.

  ssh: connect to host ippc72 port 22: No route to host
  /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 20: [: : integer expression expected
  /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 30: [: : integer expression expected
  

• CCL: 20:30 pstamp over 100k jobs --> restart

ipp@ippc19 ==> ippc76 Host key verification failed. remove known_hosts related to ippc19, i.e., remove this line "10.10.20.74 ssh-rsa xxxxxx"

ipp107 disk usage > 97% , set it to be repair to avoid data storing. neb-host --state repair --host ipp107 --note 'CCL: up->repair: disk full > 97%'

o5567g0384o	XY64	274464	1505781	update	update.LAP.PV3	LAP.PV3.20140730.ipp.20150211	LAP.ThreePi	2	PS1	GPC1	2011-01-06 15:42:26	171.377302	29.393080	ps1_11_0821	y.00000	30.00	1.0360	119.31	12.36	2176	5.38	135.3PI.y.QE-1 ps1_11_0821 visit 1
neb://ipp051.0/gpc1/20110106/o5567g0384o/o5567g0384o.ota64.burn.tbl size 0

    ipp_apply_burntool_fix.pl --exp_name o5567g0384o --class_id XY64 --verbose --dbname gpc1
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1505781	
    chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1505781

Saturday : 2019.05.04

• CCL: login to ipp113 as ippitc, but got this error

  /usr/bin/xauth:  /data/ippc64.1/ippitc/.Xauthority not writable, changes will be ignored
  /usr/bin/xauth:  /data/ippc64.1/ippitc/.Xauthority not writable, changes ignored
  
  can be solved by removing .Xauthority file
  mv .Xauthority to .Xauthority.20190504
  

• CCL: 12:00 feed update.LAP.PV3 3000 jobs

Sunday : 2019.05.05

Monday : 2019.05.06

Tuesday : 2019.05.07

Wednesday : 2019.05.08

Thursday : 2019.05.09