PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2019.01.14

• MEH: very old WW mops diffs and WS catalogs on datastore -- clean out (dsreg for WW and distribution goto_cleaned) to past couple months so page loads more easily for checking
• MEH: PS1 subset/interleaved OSS chunks interfering with normal MOPS operations -- comment string was changed from OSS.R% to EU.r%, manually alter nightly_science.config to proceed with processing
• MEH: PS2 back on sky -- WW diffs not queuing -- looks like the dark done in afternoon didn't have the expected comment of darks_afternoon?

  ENGINEERING	42707	o8498h0002d	2019-01-15 03:25:58	CLEAR.00002	30	darks_morn - Dark exp 30s 1 of 1
  
  nightly_science.pl --dbname gpc2 --camera GPC2 --verbose  --queue_diffs --date 2019-01-15
  

  • setup crontab to occasionally run this...
• MEH: PS2 WWdiffs taking very long time to run on a few skycells per diff_id -- manually killing ppSub on long running jobs and set quality 42 to allow the rest of the skycells to get to MOPS

Tuesday : 2019.01.15

• MEH: messed up PS1 EU config that blocked processing on the last OSS chunk, fixed for tonight tests okay -- new nightly processing label EU.nightlyscience added to stdscience with priority just below %SS
• MEH: PS2 WWdiffs still regularly taking >1-2hrs and delaying processing (also using excessive memory that overloads the heavily loaded nodes) -- create crontab to kill jobs after 3600s
  • manually add c2 group a 2-3x (more could also add more memory overload) to help move things along while waiting for problematic ppSub jobs to clear
  • looks like excessive detections on known odd group of cells from last Fall in chip near edge of camera -- looks like different set now, one in OTA in corner and one near middle (obvious poor cells in camera image)

Wednesday : 2019.01.16

• MEH: bad exposure (GPC1 crash, multiple ota missing) stuck in summitcopy -- o8499g0264o -- should set to drop
• MEH: another example of pantasks autorestart borking job -- summitcopy at toadvance state o8500g0002d -- manually stop & restart summitcopy again is one way to clear (and clear any possible other oddities in the pantasks that may have happened)

Thursday : 2019.01.17

• MEH: another bad/incomplete exposure in summitcopy -- o8500g0392o
• MEH: reducing time to kill long running ppSub jobs to 1200s (average is 100-200s) until poor cells can be masked

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD