== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.01.25 ===
 * 16:00 EAM: ipp007 rebooted to catch a disk, disk array rebuilding (temporarily at repair in nebulous).
 * 16:01 EAM: stopping pantasks for restart
   * 16:05 EAM: restart is done

=== Tuesday : 2016.01.26 ===
 * 02:00 MEH: looks like ipp017 is unresponsive for past 5 min, nothing on console -- set neb-host down (from repair) to prevent possible stalling for detrends, no jobs are running on it
  * try power cycle after 10 min -- back up after F1 for keyboard error -- leaving neb-host down for night
 * 02:36 MEH: ipp017 down again ~5 min ago.. setting power off for tonight 
 * 16:00 EAM : stopping the pantasks for a restart.  Haydn is working on replacement mobos for ipp017, ippx052, ipx070, all currently off.
 * 20:06 MEH: summit fault on o7414g0102o -- will finish clearing after SNIFS time finished.. dell switches had to be cycles and made the exposure appear missing and fault set -- manually removed fault once things back up and seems okay
{{{
update summitExp set fault=0 where fault=110 and exp_name="o7414g0102o";
}}}

=== Wednesday : 2016.01.27 ===
 * 14:39 MEH: boosting pstamp for QUB and then regular restart of nightly pantasks


=== Thursday : 2016.01.28 ===
 * 07:12 MEH: summitcopy and registration been having some >1200s stalled jobs occasionally lately -- manually killed to finish nightly downloads
 * 07:25 MEH: ippc09 disk full -- clear all the other ippc0x machines used for nebulous as well
 * 11:09 MEH: looks like OSS may have out-of-order diffs -- ps1_29_2690, ps1_29_2523 missing visit 1 due to FWHM>12 pix at cam stage. ps1_29_2690 has visit 2-3,3-4 but ps1_29_2523 only has visit 2-3. will wait for MOPS to ask if wanted.
 * 16:08 MEH: regular restart of nightly pantasks

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===