PS1 IPP Czar Logs for the week 2013.02.04 - 2013.02.11

(Up to PS1 IPP Czar Logs)

Monday : 2013.02.04

Tuesday : 2013.02.05

• 13:59: CZW: Moved ipp006 and ipp007 from "wave1" to "wave1_dvo" in the pantasks.hosts file. I also re-enabled the compute3 machines for stdscience. Restarting stdscience for these changes to take effect.

Wednesday : 2013.02.06

• 16:30 Serge: Queued 4 exposures for stdscience reprocessing (label: MopsSynthetics)

Thursday : 2013.02.07

• 09:25 Bill queued 5 filter staticsky runs for skycells with 4h < RA < 6h. Adding staticsky to the stack pantasks
• 10:45 Bill reverted 4 stacks that got fault 2 due to nfs issues.
• 10:50 Fixed some broken files

  recovered lost instance of o5507g0448o.ota25.fits using deneb-locate.py
  reran burntool for o5464g0516o.ota26
  recovered lost instance of o5463g0418o.ota23.fits
  reran burntool for o5170g0154o.ota32.fits 
  

• 10:50 Serge: restarted stdscience
• 14:27 Bill: set stack and stdscience pantasks to stop in preparation for load rebalancing
• 14:30 Bill: set 2 x compute3 to off in stdscience and added them to stack. Set 2 x compute2 to off in stack. Not adding them to stdscience for now since they are still overloaded with existing processing.
• 14:38 Bill fixed burntool table for o5456g0481o.ota26.fits
• 14:45 Bill is running some M31jobs on compute3 and wave4. It should only take a few minutes.
• 16:05 Bill's jobs are done. Setting 2x compute3 back to on stdscience
• 16:10 restarted distribution. The job counts were all mangled by timeouts.

Friday : 2013.02.08

• 09:45 Bill stopped stdscience. massive job failures. Looks like an nfs or nebulous problem

Saturday : 2013.02.09

Sunday : YYYY.MM.DD