PS1 IPP Czar Logs for the week 2019.05.03 - 2019.05.09

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.03

pstamp requests by JRF

Received some error emails about pstamp requests. checking out the pstamp server we can see that over 800,000 jobs have been submitted by QUB.

The errors come from a script run by the crontab on ippc26 (this is mentioned in the emails). The script checks if pstamp has over 100,000 jobs done as this is were pantasks becomes flaky.

Checking out the status of the pstamp pantasks keep an eye on:

  request.finish.run

as we DO NOT want to restart if this is running as it will cause funkiness.

So let's restart pstamp, doing the usual:

  pantasks> stop
  pantasks> shutdown now    --check that the jobs have finished first

  ~ippitc/pstamp> pantasks_server & pantasks_client
  pantasks> server input input
  pantasks> setup
  pantasks> run

I have added ippc76 to the hosts_apache group in the ~ippitc/ippconfig/pantasks_hosts.input file. This is so that when the nodes are allocated that ippc76 will be turned off when the apache group is removed.

Since over 800,000 requests were made the pstamp pantasks will need restarting about every hour or so.

• JRF: ippc72 has died, sent an email to Haydn, left the PDU off. Also commented it out of the ~ippitc/ippconfig/nebservers.csh file.

CZAR HANDOVER by JRF

• ipp139 has started having issues so there will likely be some liaising with hardware people to fix that.
• ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
• 'wait' state: the past week has seen some chip states set to 'wait' which is not a valid state. They can appear in IPP processing at times, and the quickest way to fix them is: change label + imfile update:

    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id XXXXXX
    chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id XXXXXX
  

  the update second part is needed as the change from wait to update isn't usually recognized.
• More updates will need to be queued, details of how to do this are given in the Thursday 2019-05-02 czarlog
• Guide for desperate diffs is in the czarlog for last week. HOWEVER, TdB likely has a better way of doing it by now
• QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).

CCL took over after 2pm

• CCL: need to modify /home/panstarrs/ipp/local/bin/apachedisk_chk.sh as ipp on ippc19 to solve the crontab error message, i.e., remove 72 & add 76 for host list.

  ssh: connect to host ippc72 port 22: No route to host
  /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 20: [: : integer expression expected
  /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 30: [: : integer expression expected
  

• CCL: 20:30 pstamp over 100k jobs --> restart

ipp@ippc19 ==> ippc76 Host key verification failed. remove known_hosts related to ippc19, i.e., remove this line "10.10.20.74 ssh-rsa xxxxxx"

ipp107 disk usage > 97% , set it to be repair to avoid data storing. neb-host --state repair --host ipp107 --note 'CCL: up->repair: disk full > 97%'

o5567g0384o	XY64	274464	1505781	update	update.LAP.PV3	LAP.PV3.20140730.ipp.20150211	LAP.ThreePi	2	PS1	GPC1	2011-01-06 15:42:26	171.377302	29.393080	ps1_11_0821	y.00000	30.00	1.0360	119.31	12.36	2176	5.38	135.3PI.y.QE-1 ps1_11_0821 visit 1
neb://ipp051.0/gpc1/20110106/o5567g0384o/o5567g0384o.ota64.burn.tbl size 0

    ipp_apply_burntool_fix.pl --exp_name o5567g0384o --class_id XY64 --verbose --dbname gpc1
    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1505781	
    chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1505781

Saturday : 2019.05.04

• CCL: login to ipp113 as ippitc, but got this error

  /usr/bin/xauth:  /data/ippc64.1/ippitc/.Xauthority not writable, changes will be ignored
  /usr/bin/xauth:  /data/ippc64.1/ippitc/.Xauthority not writable, changes ignored
  
  can be solved by removing .Xauthority file
  mv .Xauthority to .Xauthority.20190504
  

• CCL: 12:00 feed update.LAP.PV3 3000 jobs

Sunday : 2019.05.05

Monday : 2019.05.06

• TdB: update regarding desperate diffs and testing: We want a testcase that does not involve live testing. Therefore, make use of the daily testquad. I ran a separate reduction of the quad under the label tdbdifftest. This results in the following exposures:

  Exp Name 	Exp ID 	Chip ID Cam ID 	Fake ID Warp ID state 	label 	    data grp              Date/Time	          Object        Comment 
  o6370g0351o 	590994 	2158115 2116160 2085063 2091717 full 	tdbdifftest tdbdifftest.20190503  2013-03-19 11:37:26	  ps1_14_2124	162.OSS.O.Q.w ps1_14_2124 visit 1
  o6370g0373o 	591016 	2158118 2116155 2085069 2091722 full 	tdbdifftest tdbdifftest.20190503  2013-03-19 11:59:29	  ps1_14_2124	162.OSS.O.Q.w ps1_14_2124 visit 2
  o6370g0395o 	591038 	2158116 2116151 2085064 2091718 full 	tdbdifftest tdbdifftest.20190503  2013-03-19 12:21:39	  ps1_14_2124	162.OSS.O.Q.w ps1_14_2124 visit 3
  o6370g0417o 	591060 	2158117 2116154 2085067 2091720 full 	tdbdifftest tdbdifftest.20190503  2013-03-19 12:43:52	  ps1_14_2124	162.OSS.O.Q.w ps1_14_2124 visit 4
  

  We can then make the regular diffs with those:

  difftool -dbname gpc1 -definewarpwarp -warp_id 2091717 -template_warp_id 2091722 -backwards  -set_workdir neb://@HOST@.0/gpc1/tdbdifftest.20190503 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun
  difftool -dbname gpc1 -definewarpwarp -warp_id 2091718 -template_warp_id 2091720 -backwards  -set_workdir neb://@HOST@.0/gpc1/tdbdifftest.20190503 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun
  

  resulting in the following exposures:

  diff_ID state label data grp dist grp Magicked tess ID both ways Time registered reduction workdir
  1759513 full tdbdifftest tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:41 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503         
  1759514 full tdbdifftest tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:42 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503  	
  

Tuesday : 2019.05.07

Wednesday : 2019.05.08

Thursday : 2019.05.09