== PS1 IPP Czar Logs for the week 2018.04.30 - 2018.05.06 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2018.04.30 ===
  * 14:00 : spent part of the morning clearing pstamp failures.  These had three types of problems: 1) chip imfiles in state = error_clean, 2) chip imfiles missing PSF files (stuck only on stsci nodes), 3) warp skyfiles in state error clean.  I used these types of commands to clear
{{{
chiptool -dbname gpc1 -updaterun -chip_id $chip_id -set_state cleaned
chiptool -dbname gpc1 -setimfiletoupdate -chip_id $chip_id -set_label ws_nightly_update -set_update_mode 1
warptool -dbname gpc1 -updaterun -warp_id $warp_id -set_state cleaned
warptool -dbname gpc1 -setskyfiletoupdate -warp_id $warp_id -set_label ws_nightly_update
neb-mv FOO.psf FOO.psf.bad
}}}
  * 15:15 CZW: I have updated the ippps2 nightly_science.config to remove all nightly_science targets.  This should prevent any training/testing data from being interpreted as real science data.  I have also updated the camera format to use the FILTERID header to set the FPA.FILTER concept, which will allow registration to insert full filter names.
  * 15:30 CZW: ippb02 rsyncs are stopped so I can transfer data from the .0 partitions of ippb07/8 to the .1 partitions, because I wrote the commands wrong.
  * 16:30 CZW: The ingestion of GPC1 stacks into the gpc2 database cannot be completed due to a lack of usable w science data from GPC2.  This is needed to construct the reference warp that links the stackRun entry to the rawExp table.  Making a note here so I don't forget to fix this when GPC2 w data exists.
 * MEH: reprocessed the remaining 2 exposures still broken from before (o7023g0257o, o7023g0317o) so MOPS can get their data as soon as needed without further delays -- looks like the previous fix was still in an odd not-updated state in warp that pstamp could use, a normal cleanup cycle seems to have cleared and now those exposures with the broken LAP label can fully update. 

 * MEH: restored /export/ippx029.0/ipp dir after Haydn rebuilt the raid -- needs to be tested in processing
 * MEH: ganglia not reporting on ippx071, ippc127, ippdb03, ippc17, ippc27, ippx052 for well over a week now -- restarted gmond and seem fine now -- if not reporting then cannot see problems, czars need to be regularly checking this
 * MEH: the daily pantasks restart seems to be happening again now
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===