PS1 IPP Czar Logs for the week 2018.04.30 - 2018.05.06

(Up to PS1 IPP Czar Logs)

Monday : 2018.04.30

• 14:00 : spent part of the morning clearing pstamp failures. These had three types of problems: 1) chip imfiles in state = error_clean, 2) chip imfiles missing PSF files (stuck only on stsci nodes), 3) warp skyfiles in state error clean. I used these types of commands to clear

  chiptool -dbname gpc1 -updaterun -chip_id $chip_id -set_state cleaned
  chiptool -dbname gpc1 -setimfiletoupdate -chip_id $chip_id -set_label ws_nightly_update -set_update_mode 1
  warptool -dbname gpc1 -updaterun -warp_id $warp_id -set_state cleaned
  warptool -dbname gpc1 -setskyfiletoupdate -warp_id $warp_id -set_label ws_nightly_update
  neb-mv FOO.psf FOO.psf.bad
  

• 15:15 CZW: I have updated the ippps2 nightly_science.config to remove all nightly_science targets. This should prevent any training/testing data from being interpreted as real science data. I have also updated the camera format to use the FILTERID header to set the FPA.FILTER concept, which will allow registration to insert full filter names.
• 15:30 CZW: ippb02 rsyncs are stopped so I can transfer data from the .0 partitions of ippb07/8 to the .1 partitions, because I wrote the commands wrong.
• 16:30 CZW: The ingestion of GPC1 stacks into the gpc2 database cannot be completed due to a lack of usable w science data from GPC2. This is needed to construct the reference warp that links the stackRun entry to the rawExp table. Making a note here so I don't forget to fix this when GPC2 w data exists.

• MEH: reprocessed the remaining 2 exposures still broken from before (o7023g0257o, o7023g0317o) so MOPS can get their data as soon as needed without further delays -- looks like the previous fix was still in an odd not-updated state in warp that pstamp could use, a normal cleanup cycle seems to have cleared and now those exposures with the broken LAP label can fully update.

• MEH: restored /export/ippx029.0/ipp dir after Haydn rebuilt the raid -- needs to be tested in processing
• MEH: ganglia not reporting on ippx071, ippc127, ippdb03, ippc17, ippc27, ippx052 for well over a week now -- restarted gmond and seem fine now -- if not reporting then cannot see problems, czars need to be regularly checking this
• MEH: the daily pantasks restart seems to be happening again now

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD