== PS1 IPP Czar Logs for the week 2019.01.07 - 2019.01.13 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2019.01.07 ===
 * CCL: note from [wiki:PS1_IPP_Czarlog_20181119], pstamp jobs stuck >7 days
{{{
pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912774
pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912781
pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912828
}}}

=== Tuesday : 2019.01.08 ===

=== Wednesday : 2019.01.09 ===

=== Thursday : 2019.01.10 ===
 * MEH: removed pstamp from auto-restart to prevent stamp requests still parsing from being corrupted, will restart pstamp once the stamp requests have finished
=== Friday : 2019.01.11 ===
 * MEH: Serge is testing out running some jobs on the IPP nodes (ipps, ippx, ippx35-63) when nightly processing has finished
  * nebulous db (ippdb09) not responding well, adjustments being made
=== Saturday : 2019.01.12 ===
 * MEH: ippc70 crashed last night -- power cycle not bringing back online, power off until can be looked at -- comment out in ippconfig/nebservers.csh (in ~ippitc, ~ippps2, ~ipp) and change NEB_HOST to ippc71.. restart pantasks in fresh login just to be sure
 * MEH: nagios should have alerted us, Gavin found emails being held moderator since 12/10 -- large backlog of email now available
 * MEH: ippc70 crash wedged registration -- 
{{{
select * from rawImfile where exp_id>=1440841 and data_state not like "full";

| exp_id  | class_id | data_state       |
+---------+----------+------------------+
| 1440841 | XY45     | check_burntool   | 
| 1440842 | XY45     | pending_burntool | 
....

regtool -updateprocessedimfile -exp_id 1440841 -class_id XY45 -set_state pending_burntool -dbname gpc1

}}}
=== Sunday : 2019.01.13 ===