| 1 | | test |
| | 1 | == Processing Throughput Plots == |
| | 2 | |
| | 3 | I plotted the things processed/hour for a number of stages, for the month of September 2010, to look for patterns in the processing. These plots include any kind of processing (stdscience, update, pstamp, etc). I couldn't really find any patterns, other than that we did more processing on 24-25 (photfest), and there was a pile of data that went through chip/cam/warp on 8 & 9 (no idea what this is). |
| | 4 | |
| | 5 | |
| | 6 | == Grabbing the data == |
| | 7 | |
| | 8 | Grabbing the data to plot the throughput was a bit complicated. Not all the stages have times in them, and some of them that do have inconsistent time zones, or are not used at all. |
| | 9 | |
| | 10 | stages with no columns relating to times |
| | 11 | * chipRun |
| | 12 | * camRun |
| | 13 | * magicDSRun |
| | 14 | |
| | 15 | stages with incomplete time information |
| | 16 | * warpRun (all values since 2008 are NULL) |
| | 17 | |
| | 18 | stages with epoch (timestamp) time information |
| | 19 | * summitExp |
| | 20 | * newExp |
| | 21 | * rawExp |
| | 22 | * fakeRun |
| | 23 | * flatcorrRun |
| | 24 | |
| | 25 | stages with registered (with time set by a script) time information |
| | 26 | * stackRun |
| | 27 | * diffRun |
| | 28 | * magicRun |
| | 29 | * chipBackgroundRun |
| | 30 | * detRun |
| | 31 | * diffPhotRun |
| | 32 | * dqstatsRun |
| | 33 | * staticskyRun |
| | 34 | |
| | 35 | stages with time_stamp (timestamp) |
| | 36 | * distRun |
| | 37 | |
| | 38 | I did not use all of these stages/tables. For the stages that I used, I discovered that epoch uses HST, and registered uses UTC. |
| | 39 | |
| | 40 | chip, cam, warp were stages with missing information. To get this information, I grabbed a list of chips with chip_id > some_number, and did neb-stat on the log file to find out when it was created. I did similar queries for cam and warp. |
| | 41 | |
| | 42 | For stack, diff, magic, registered refers to when they were added to the db to be processed, not when they were processed. This is why there are spikes for stacks, they are all queued up at the same time each day. |
| | 43 | |
| | 44 | == The plots == |
| | 45 | |
| | 46 | Color code |
| | 47 | |
| | 48 | * red - epoch (in gpc1 db, it is a TIMESTAMP) |
| | 49 | * cyan - file creation times were grabbed from the log files (slow). |
| | 50 | * purple - registered (in gpc1 db) |
| | 51 | * black - time_stamp (in gpc1 db, it is a TIMESTAMP) |
| | 52 | |
| | 53 | Black vertical lines are for 6am HST for each day. |
| | 54 | The numbers on the bottom refer to the date. |
| | 55 | |
| | 56 | [[Image(processing_thru-0.png)]] |
| | 57 | |
| | 58 | [[Image(processing_thru-1.png)]] |
| | 59 | |
| | 60 | [[Image(processing_thru-2.png)]] |
| | 61 | |
| | 62 | * I haven't finished grabbing all chip and warp dates for September... |
