PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.15

• 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again
• 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
  

• 14:00 CZW: Restarting stdlocal. I'm going to see what happens with the "default" set of hosts.
  • 14:07 CZW: default set is underpowered. Turning on xnodes with xnode.hosts.on macro.
• 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
• 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
  

• 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  • back online -- setting to repair only so not to overload until we catchup
• 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way
• 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing. I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
• 16:25 CZW: Restarting the ipp user pantasks. I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD