PS1 IPP Czar Logs for the week 2011-09-12 - 2011-09-18

(Up to PS1 IPP Czar Logs)

Monday : 2011-09-12

• Around 10am Bill finally got all of the pieces checked into the branch for the ppMops memory reduction fix. pantasks were restarted
• 11:00 adjusted pstamp.dependent.run task to not be so agressive at running. Should reduce the database load that it cause.
• 15:00 CZW: Reworked host definitions to be more equitable (hopefully) and to run the processors we have as hard as possible without crashing anything. New definitions for all servers are stored in /home/panstarrs/ipp/ippconfig/pantasks_hosts.input

Tuesday : YYYY.MM.DD

• 09:50 Mark (czar): removed from nebulous and processing stopped on ipp021 for Cindy to upgrade the motherboard.
• 13:00 ipp021 back online. Added back nebulous and pantasks.
• 15:10 ipp026 went down. Removed from nebulous list until back up. Kernel panic similar to what has happened before, Chris rebooted and added details to ipp026-crash-20110913. Put back into nebulous in same state as before: repair.
• 21:30 registration trouble, in pantasks.stdout.log and run manually

  crash for: ipp_apply_burntool_single.pl --exp_id 392018 --class_id XY04 --this_uri neb://ipp006.0/gpc1/20110914/o5818g0058o/o5818g0058o.ota04.fits --continue 10 --previous_uri neb://ipp006.0/gpc1/20110914/o5818g0057o/o5818g0057o.ota04.fits --dbname gpc1 --verbose
  

Wednesday : YYYY.MM.DD

• 06:00 Mark: stalled at o5818g0448o with check_burntool and ota27
  • ipp018 was having trouble connecting to ipp007.0
• 11:45 excessive CPU use by distribution pcontrol, restarted distribution.
• 12:00 diff faulting from error reading FITS file /data/ipp042.0/nebulous/48/67/1297415775.gpc1:ThreePi.nt:2011:09:14:o5818g0508o.392470:o5818g0508o.392470.wrp.254273.skycell.2361.049.mask.fits. Regenerated with

  perl ~ipp/src/ipp-20110622/tools/runwarpskycell.pl --warp_id 254273 --skycell_id skycell.2361.049 --redirect-output 
  

• 12:10 diffim (diff_id=165304) running on ipp026 for 46ks, ppSub hanging. Killed ppSub to fault and revert. ipp026 has had timeouts to ippb00,01,02 in the past (seen in dmesg, not sure when). Diff completed.
• 12:21 Serge: stopped pstamp; dumped ippRequestServer to /export/ippc17.0/ipp/mysql-dumps/ippRequestServer.20110914.sql ; all done in less than 2 minutes. Master coordinates: mysqld-bin.000610, 505678724.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD