PS1 IPP Czar Logs for the week 2011.08.08 - 2011.08.14

(Up to PS1 IPP Czar Logs)

Monday : 2011.08.08

Tuesday : 2011.08.09

• 08:00: Roy: Registration stuck as usual. regpeek tells me offending exposure is o5782g0063o
• 08:10: Bill and Heather: stopped summitcopy pantasks, waited for Nrun column to reach zero then ran:

copy.reset

this reset the book with

book init pzPendingImfile

• 10:38 Serge: Reverted failed publishing exposures
• 11:40 Bill: changed label on runs with data_group STS.201104rerun from STS.nightlyscience to 201104rerun. (Added the label to the usual places.
• 12:03 Bill: queued STS diffs from 2011-08-06. I really need to automate this.
• 14:19 Bill: reverted cam_id 245246 which has faulted twice for "unable to access file on ipp020" for two different files. There are diff faults for files on that node as well.
• 14:25: Roy: processing slow, so stopping stdscience and...
• 14:40: Roy: ...restarted again
• 15:20: Roy: reverted stuck chips with:

camtool -revertprocessedexp -fault 3 -label ThreePi.nightlyscience -dbname gpc1

• 15:40: Roy: ippc11 is down. Tried to reboot with serial console:

 ssh rhenders:ippdb03@cab6con.ipp.ifa.hawaii.edu

but blocked by another user

• 22:30 : eam : ipp014 crashed so I rebooted it. I suspected that the ippc11 crashes may have left some things stuck, so I restarted the full system.

Wednesday : 2011.08.10

• 03:30 eam : summit copy failure. here is the error that occurred, leaving behind a crashed entry:

  Starting script /home/panstarrs/ipp/psconfig//ipp-20110622.lin64/bin/summit_copy.pl on ipp041
  
  Running [/home/panstarrs/ipp/psconfig/ipp-20110622.lin64/bin/dsget --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5783g0268o/o5783g0268o32.fits --filename neb://ipp021.0/gpc1/20110810/o5783g0268o/o5783g0268o.ota32.fits --compress --bytes 52842240 --nebulous --md5 26c60c4a3177cd43bcb803ca44e2ace5 --timeout 600 --copies 2]...
  downloading file to /tmp/o5783g0268o.ota32.fits.uznkmvcY.tmp
  can't sync nebulous filehandle: Input/output error at /home/panstarrs/ipp/psconfig/ipp-20110622.lin64/bin/dsget line 207.
  Unable to perform dsget: 5 at /home/panstarrs/ipp/psconfig//ipp-20110622.lin64/bin/summit_copy.pl line 236.
  

• 08:00 Roy: All data downloaded from last night.
• 09:10 Serge: reverted failed publishing
• 09:35 Roy: killed some chip_imfile.pl instances that had been running for too long
• 10:40 Roy: warps stuck, so ran warp.reset in stdscience

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD