PS1 IPP Czar Logs for the week 2015.01.12 - 2015.01.18

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.12

• 00:40 MEH: clearing a fault 5 diff

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 635299 -skycell_id skycell.1607.047  -dbname gpc1
  

• 00:50 MEH: stdsci poll still low, restarted
• 01:00 MEH: stdlocal set.poll 100 in case do catch up, nightly has priority and be finished in time for fiber switchover
• 02:00 MEH: ipp095 overloaded a bit again, to repair
• 06:57 Bill: two warps were repeatedly getting fault 4 from the "cannot compute curve of growth psf is invalid everywhere" assertion. Set quality for them

  warptool -set_quality 42 -fault 0 -updateskyfile -warp_id 1345413 -skycell_id skycell.0509.081
  warptool -set_quality 42 -fault 0 -updateskyfile -warp_id 1345472 -skycell_id skycell.1084.070
  

• 07:00 Bill ipp090 and ipp093 have rather high loads as well.
  • but processing rate was generally okay
• 10:20 MEH: clearing some fault 5 WSdiffs

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 635300 -skycell_id skycell.1607.038  -dbname gpc1
  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 635300 -skycell_id skycell.1607.047  -dbname gpc1
  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 635450 -skycell_id skycell.1247.076  -dbname gpc1
  

• 10:30 MEH: clearing some old WSdiff that won't complete due to old and image cleanup.. -- when time may be able to update warps to complete but...

  difftool -updaterun -diff_id 634284 -dbname gpc1 -set_label ThreePi.WS.nightlyscience.missed
  difftool -updaterun -diff_id 621228 -dbname gpc1 -set_label OSS.WS.nightlyscience.missed
  

  • did manual chip+warp updates when necessary, were actually from fault 5 and need to have quality set so now remaining diff should be released ok

    difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 634284 -skycell_id skycell.0679.021  -dbname gpc1
    difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 621228 -skycell_id skycell.1529.088  -dbname gpc1
    

  • chips for 1/10 were cleaned up already, but not warps
• 11:50 MEH: starting stdlocal chip only test for 10G link usage, stopping all other processing since will need cleared for fiber changes in couple hours anyways
  • poll~100 chip only uses ~4Gbits/s both in and out on link
  • poll~150 uses ~6-7Gbits/s both in and out on link
  • nightly overall uses ~5-7Gbits/s mostly 6-7G out and 5-6G in -- so even 150 w/ stdlocal was a bit high during nightly
• 11:55 MEH: ippc11 screen issue,

  There are screens on:
  	13722.robo	(Detached)
  	12542.RoboCzar	(Dead ???)
  	19893.CzarPoll	(Dead ???)
  	13565.plot	(Detached)
  Remove dead screens with 'screen -wipe'.
  

  • wiped, cleared other names and restarted screens
• 14:40 MEH: possible changes to network plans -- setting stdlocal back to full poll until nightly starts
• 19:40 MEH: nightly starting, stdlocal set.poll 100 -- seems to be just at limit, maybe a slow slippage in nightly chip processing which will eventually turn chip-warp off in stdlocal. ~1300 camera entries so not having stdlocal auto-off would be good.. so set.poll 80

Tuesday : 2015.01.13

• 07:45 MEH: seems stdlocal set.poll~80 is a limit, still caused enough of a backup until about 2am when stdlocal triggered auto-stop chip-warp, stdsci processing mostly kept up however, including registration. stdlocal set.poll back to 400 ~0830
• 09:10 MEH: restarting pstamp
• 10:40 MEH: processing to stop -- going to start modifying systems
  • decomission jaws nodes -- move pantasks to ippc01-c09 --
  • ippx037-x044 + 4849 switch to n5k
  • stsci ip to .30. network
  • redistribute other ippx across multiple 4849 on 6509e
• 15:45 MEH: stsci IP moved to .30. net, x037-x044 moved to nk5 and .20. net, all pantasks moved off jaws nodes -- Gene gave all clear for starting nightly pantasks to be ready for data tonight
  • while processing down, going to rotate the apache ippc0x logs --

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD