PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

• 07:20 MEH: like Saturday mornning, looks like 3 camRuns are stalling due to missing chipRun files -- are these not being checked for validity?

   	Exp Name 	Exp ID 	Chip ID 	Cam ID 	state 	label 	data grp 	dist grp 	Fault 	Revert Command
  
   	o6586g0074o 	667195 	902792 	873445 	new 	ThreePi.nightlyscience 	ThreePi.20131021 	ThreePi 	3 	camtool -revertprocessedexp -fault 3 -label ThreePi.nightlyscience -dbname gpc1
  	o6586g0398o 	667519 	903082 	873737 	new 	ThreePi.nightlyscience 	ThreePi.20131021 	ThreePi 	3 	camtool -revertprocessedexp -fault 3 -label ThreePi.nightlyscience -dbname gpc1
  	o6586g0458o 	667579 	903139 	873793 	new 	ThreePi.nightlyscience 	ThreePi.20131021 	ThreePi 	3 	camtool -revertprocessedexp -fault 3 -label ThreePi.nightlyscience -dbname gpc1 
  
  neb-less neb://any/gpc1/ThreePi.nt/2013/10/21//o6586g0074o.667195/o6586g0074o.667195.cm.873445.log
  
  Unable to parse camera.
   -> pmConfigConvertFilename (pmConfig.c:1802): System error
       Unable to access file neb://ipp041.0/gpc1/ThreePi.nt/2013/10/21//o6586g0074o.667195/o6586g0074o.667195.ch.902792.XY62.b1.fits: nebclient.c:535 nebFind() - no instances found
   -> readPHUfromFilename (pmFPAfileDefine.c:454): System error
       Failed to convert file name neb://ipp041.0/gpc1/ThreePi.nt/2013/10/21//o6586g0074o.667195/o6586g0074o.667195.ch.902792.XY62.b1.fits
   -> fpaFileDefineFromArray (pmFPAfileDefine.c:641): System error
       Failed to read PHU for neb://ipp041.0/gpc1/ThreePi.nt/2013/10/21//o6586g0074o.667195/o6586g0074o.667195.ch.902792.XY62.b1.fits
   -> ppImageDefineFile (ppImageDefineFile.c:17): unknown psLib error
       failed to load file definition ARG LIST
   -> ppImageParseCamera (ppImageParseCamera.c:12): I/O error
       Can't find an input image source
  
  perl ~ipp/src/ipp-20130712/tools/runchipimfile.pl --chip_id  902792  --class_id XY62 --redirect-output
  
  neb://any/gpc1/ThreePi.nt/2013/10/21//o6586g0398o.667519/o6586g0398o.667519.cm.873737.log
  perl ~ipp/src/ipp-20130712/tools/runchipimfile.pl --chip_id 903082   --class_id XY62 --redirect-output
  
  neb://any/gpc1/ThreePi.nt/2013/10/21//o6586g0458o.667579/o6586g0458o.667579.cm.873793.log
  perl ~ipp/src/ipp-20130712/tools/runchipimfile.pl --chip_id 903139   --class_id XY62 --redirect-output
  

• 12:05 MEH: difficult to get even status from stdsci.. time for regular restart..
  • MD03.pv2 label out for a bit for LAP to push through to stacks -- many (>400) warp skyfiles faulting, 4 camera faults to look into

• 12:30 CZW: The excessive failures of warp were due to ipp031 not having a proper /local/ipp/gpc1/tess/ hierarchy. rsync -auv /data/ipp044.0/ipp/gpc1/tess/ . seems to have resolved this problem. Reverting the failures, and seeing if they pop up again.
• 13:10 MEH: something still messed up with many (10%?) warp skyfile faults, set lap.cleanup.off to not allow more chips to trigger but will push through all the warps possible to diagnose
• 16:50 MEH: ipp041 out of processing fully, see if that helps the odd file missing faults (maybe clear up nightly processing faults as well?)
• 16:53 Bill: reran class_id XY62 for STS chip_ids 900631 900863 900954 901070 901994 whose contents were wiped out by a duplicate run for the job. The jobs all ran on ipp041 which has been a problem child. Reverted the associated camRuns

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD