| | 1 | == Processing Throughput Tiger Team Notes == |
| | 2 | |
| | 3 | === Action items from 2011.08.29 === |
| | 4 | |
| | 5 | * Chris |
| | 6 | * tidy / finish disk usage stuff (regular job to be run on all storage nodes) |
| | 7 | * LAP / magic bug : magic does not queue multiple runs for the same label & exp_id |
| | 8 | * LAP / smf destreak bug : warp-stack runs destreak the same exp_id in multiple cam_ids; these should be merged somehow (either smf level or streak level) |
| | 9 | * apache targets via ipp/.cshrc |
| | 10 | * ROC (at low priority) |
| | 11 | * Bill |
| | 12 | * streaksremove / ppMops table memory fix |
| | 13 | * log file rename |
| | 14 | * smf in pstamp / web interface |
| | 15 | * event monitoring |
| | 16 | * Mark |
| | 17 | * most large-count bad diffs had telescope tracking glitch / optical aberration |
| | 18 | * examine diff stats from moments |
| | 19 | * other ways to automatically reject these? |
| | 20 | * MD02 starting |
| | 21 | |
| | 22 | === Notes from 2011.08.25 === |
| | 23 | |
| | 24 | * Bottlenecks |
| | 25 | * Need to catch files with storage object but no instance. |
| | 26 | * Bill says that this "should" only be an issue with log files. Suggestion for that is to move old logfile to logfile.timestamp |
| | 27 | * That way repeated failures can be diagnosed (editors note: somewhat but not in ippMonitor ....) |
| | 28 | * Long running jobs |
| | 29 | * Need a way to have pantasks stop and fault jobs that are stuck due to |
| | 30 | * nfs hangs |
| | 31 | * alogrithm problems (should not be re-run) |
| | 32 | * Talked about the "Event monitoriing" hooks that Bill was working on. |
| | 33 | * When a nebulous apache servers does not respond it causes problems throughout the system because the requests don't get serviced |
| | 34 | * Suggestion to investigate more robust load balancing solution. Apache, hardware solutions, etc |
| | 35 | * short term perhaps target apache requests to a server based on the client node. That way errors would be localized to a subset of machines |
| | 36 | * ipp014 |
| | 37 | * New RAID card on order. Turned off as processing node and for nebulous output (repair mode) |
| | 38 | * memory usage |
| | 39 | * Mark is investigating ppMops explosion (related to diff source explosion) |
| | 40 | * Bill is investigating higher than expected memory usage by streaksremove |
| | 41 | |
| | 42 | * Disk space |
| | 43 | * We are still using more space than predicted. Chris has scripts running to evaluate. |
| | 44 | |
| | 45 | * What about long term storage on data store. Do we need to keep the clean camera and diff runs forever? |
| | 46 | * Bill created a web interface to get the smfs for Dave Monet to use from Manoa. Can this be hardened and put on the pstamp server? |
