IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 1 and Version 2 of PS1_IPP_Czarlog_20190510


Ignore:
Timestamp:
May 10, 2019, 2:40:25 PM (7 years ago)
Author:
tdeboer
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20190510

    v1 v2  
    1717 - these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh
    1818
     19 * TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues:
     20{{{
     21mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcsh
     22}}}
     23 I took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.
     24
    1925=== Saturday : 2019.05.11 ===
    2026