IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 25 and Version 26 of Production_Work_Log


Ignore:
Timestamp:
Nov 10, 2011, 3:24:31 PM (15 years ago)
Author:
huntley
Comment:

Reversed the order of 2011's entries, so they are all most recent first.

Legend:

Unmodified
Added
Removed
Modified
  • Production_Work_Log

    v25 v26  
    22
    33= 2011 Production Node Work =
    4 == Jan 21 ==
    5 replaced motherboard hardware and disks for ipp009, ipp012, ipp014.
    6 
    7 ipp012 new disk 11 had to be swapped out.
    8 ipp009: 6 4BG sticks were installed but only 5 are showing.  All the memory has been swapped out for new memory with the same result.
    9 ipp014: We hear continuous beeps on power on changing too 2 short
    10 beeps and a long one after something appears on the screen.  However
    11 the 2 beeps and a long on are not constant - they become random.
    12 We don't see any memory report on the first screen like the others.
    13 We changed the mother board and memory with identical results.
    14 There was a dead fan issue but we rewired it and all the fans are
    15 working and the fan light is off.  No effect on the beep pattern. The
    16 initial quick short beeps sound like memory beeps but it weird that we
    17 get it to boot at all. Left out of the rack.
    18 
    19 The new mother boards do not have an IDE connection
    20 for the existing optical drives.  We (Bill) drove to the only 2
    21 computer stores on the island and purchased the only 2 notebook size
    22 optical drives with sata connectors available. However - once we
    23 installed these we realized the connectors were something we had not
    24 seen - and did not have the cables for. At first glance they look like
    25 sata connections but there is a power component as well.  opps.
    26 Fortunately Brad had external usb optical drives we could use.
    27 Hundreds of them, in fact.  Gavin had to Mac Gyvered them in place
    28 because the cables were too short to set them anywhere.
     4
     5== Nov 7 ==
     6 * team racking and installation of 13 5 U's
     7 * put cpu back into ipp036
     8 * put ippc10 back in rack
     9 
     10== Nov 4 ==
     11 * ippb00 Haydn replaced back plane for ippb00 a0 - Drive 3 detected. All drives put back in unit - parts sent back to YMI.
     12
     13== Nov 1 ==
     14 * Installed ASUS MOBO into ippc10
     15
     16== Oct 27 ==
     17 * Installed spare ASUS MOBO into ippc10 - cpu heat sinks do not install properly - Brought ippc10 back to ATRC -ordered proper CPU carriage frame from YMI
     18
     19== Oct 25 ==
     20 * ippb00 replaced cable,raid card and sled - still not detecting drive A0 3 - ordered back plane
     21
     22== Oct 24 ==
     23 * ipp036 has dead CPU0 fan  unable to boot.
     24 * Swapped MOBO out of ippc10 into ipp036 (still in rack - unable to remove it - only me working ) Left ipp036 with 1 processor - not sure if cpu damaged - did want to hurt MOBO of running system
     25 * Installed 6 of the 8 3TB drives in STARE nodes. 2 drive DOA sent for RMA
     26
     27== Oct 19 ==
     28 * replaced ippb00 A0 drive 3 but still not being detected. worked with LSI to de-bug - needs hardware fix Ordered parts from YMI
     29
     30== Oct 10 ==
     31 * Swapped CPU0 memory with CPU1 memory
     32
     33== Sept 30 ==
     34 * installed 3TB disks in slots 2&3
     35
     36== Sept 20 ==
     37 * Bill and Haydn Installed the new MOBO on ipp020
     38
     39== Sept 13 ==
     40 * Bill and Haydn Installed the new MOBO on ipp021
     41
     42== Aug 31 ==
     43 * ipp010, ipp017, ipp020 (disk upgrades) and ipp021 (mobo
     44upgrade).
     45
     46== Aug 30 ==
     47 * Replaced IPP014's raid controller.
     48
     49== Aug 23 2011 ==
     50 * Haydn Bill and I were able upgrade ipp004,ipp013 and ipp018 MB and disks without incident.
     51 * We corrected the existing error in the RAID card cabling in ipp004 so now the drives are seated in the same order as the other machines ( funnily we discovered and remembered this problem because we had missed swapping a drive and put a 500GB back in which was showing in the wrong slot )
     52
     53 * ipp037 has a new MB.
     54
     55 * Gavin installed the OS and building the RAID - he was instrumental once again in supporting us while at MHPCC.
     56
     57 * Haydn did all the drive replacements
     58
     59== Aug 16 ==
     60 * Installed the replacement MOBO for ippc10
     61
     62== Aug 15 ==
     63 * ipp011
     64   * Upgraded to new MB and 2 TB drives. Had to move CPU0 memory stick from outer blue slot to middle black slot.
     65   * Suppose to fill blue slots first. The memory was not detected in the outer black slot. Same weirdness seen in ipp009 ( according to Gavin ASA chassis)
     66
     67 * ipp019
     68   * Upgraded to new MB and 2 TB drives. Memory all in blue slots and detected
     69   * The 1st new MB installed did not detect any memory in the cpu0 no matter where we put the sticks. Tried a second new MB using all the same memory and CPU's - with good results.  Will RMA the 1st new MB.
     70
     71 * ipp015
     72   * Upgraded to 2TB drives.  On reboot saw only 12GB of memory.  For this machine memory is suppose to be an all black slots.  However we noted that the memory was in a mixture of black and blue slots. Moved to all blacks. All memory detected.
     73
     74== Aug 1 ==
     75 * Swapped all memory out of ippdb03 - formally ippc00 - using the memory from ippc10  - waiting for MB
     76 * Added 2 2TB disks to ippc10 to become the newest rendition of ippdb03
     77
     78== July 29 ==
     79 * Added new memory to ippb00 1nd 01 - now have 32GB each.
     80
     81== July 27 ==
     82 * Spent AM diagnosing the non-boot problem in ipp030 - failed MB
     83 * After lunch 
     84 * Took failed MB out of ipp030 and replaced it with MB from ippc10.
     85 * ippc10 remains out of the rack.
     86
     87== July 15 ==
     88 * Swapped the 1st 3 mem sticks out of ippdb00
     89
     90== May 16 ==
     91 * replace a raid card in ipp005
     92
     93== Feb 9 ==
     94 * Bill and I replaced the motherboards of ipp005, ipp006, ipp007, ipp025
     95
     96Notables:
     97ipp006 & ipp025 memory counts were missing 4GB.  We identified which
     98slot and put the stick in a black slot next to a blue working slot.
     99All 24GB are now showing on both machines. The malfunctioning slots on
     100the 2 machines were not the same. We tested the power cables and while
     101we found some odd readings they were not consistent between the
     102machines or relative to the cpu's with the malfunctioning slots.
     103
     104The boards installed on the back of the CD drive are not working.  We
     105still have external CD drives hanging off the machines.
     106We need to come up with a solution.
     107
     108At some point in time - this fix should work for ipp009 so we have all
     10924 GB available.
     110
     111ipp006:
     112Voltage readings for CPU0 8pin power connector:
     113+5v   +12v   +3.3V
     1145.2   12.5    3.4
     115
     116-12V    +12V   5vSB      P6
     11711.9    12.2*   5.4*   410ms
     118
     119* blinking
     120
     121Voltage readings for CPU1 8pin power connector:
     122same as cpu0 except for pg was 590ms and blinking too.
     123
     124CPU1 slot closest to the cpu was not working.
     125
     126ipp025:
     127Voltage readings for CPU0 8pin power connector:
     128Same display for ipp006 24 pin and 8 pin cpu0
     129
     130Voltage readings for CPU0 8pin power connector:
     131Same for CPU1 8pin.  pg 430ms
     132
     133== Feb 4 ==
     134
     135We installed the new RAID card in ipp012 without problems.
     136
     137However, we did not have success with the IDE->SATA converter board
     138for the CD/DVD. The drive was not being detected on boot. We tried all
     139SATA ports, different card, different CD/DVD, different cable to no
     140avail.  We checked the bios and there was note anything we could see
     141that would prevent it from working.  The external drive has been
     142reattached. We brought a card and cd/dvd player back to test it and
     143get it working here. It may need a jumper set to make sure it is not
     144in slave mode - however we did not have a manual or a jumper - the
     145boards did not come with one.
     146
     147We also had the usual rack problems - ipp012 is particularly difficult
     148to put back in - it is at the bottom of the rack and ipp013 is
     149weighting down on it a bit.
     150
     151== Jan 31 ==
     152disk upgrades on ipp008, ipp016, and ipp021, and motherboard
     153upgrades on ipp008 and ipp016.
    29154
    30155== Jan 24 ==
     
    47172memory problem occurred with the good power supplies as well.
    48173
    49 
    50 == Jan 31 ==
    51 disk upgrades on ipp008, ipp016, and ipp021, and motherboard
    52 upgrades on ipp008 and ipp016.
    53 
    54 == Feb 4 ==
    55 
    56 We installed the new RAID card in ipp012 without problems.
    57 
    58 However, we did not have success with the IDE->SATA converter board
    59 for the CD/DVD. The drive was not being detected on boot. We tried all
    60 SATA ports, different card, different CD/DVD, different cable to no
    61 avail.  We checked the bios and there was note anything we could see
    62 that would prevent it from working.  The external drive has been
    63 reattached. We brought a card and cd/dvd player back to test it and
    64 get it working here. It may need a jumper set to make sure it is not
    65 in slave mode - however we did not have a manual or a jumper - the
    66 boards did not come with one.
    67 
    68 We also had the usual rack problems - ipp012 is particularly difficult
    69 to put back in - it is at the bottom of the rack and ipp013 is
    70 weighting down on it a bit.
    71 
    72 == Feb 9 ==
    73 Bill and I replaced the motherboards of ipp005, ipp006, ipp007, ipp025
    74 
    75 Notables:
    76 ipp006 & ipp025 memory counts were missing 4GB.  We identified which
    77 slot and put the stick in a black slot next to a blue working slot.
    78 All 24GB are now showing on both machines. The malfunctioning slots on
    79 the 2 machines were not the same. We tested the power cables and while
    80 we found some odd readings they were not consistent between the
    81 machines or relative to the cpu's with the malfunctioning slots.
    82 
    83 The boards installed on the back of the CD drive are not working.  We
    84 still have external CD drives hanging off the machines.
    85 We need to come up with a solution.
    86 
    87 At some point in time - this fix should work for ipp009 so we have all
    88 24 GB available.
    89 
    90 ipp006:
    91 Voltage readings for CPU0 8pin power connector:
    92 +5v   +12v   +3.3V
    93 5.2   12.5    3.4
    94 
    95 -12V    +12V   5vSB      P6
    96 11.9    12.2*   5.4*   410ms
    97 
    98 * blinking
    99 
    100 Voltage readings for CPU1 8pin power connector:
    101 same as cpu0 except for pg was 590ms and blinking too.
    102 
    103 CPU1 slot closest to the cpu was not working.
    104 
    105 ipp025:
    106 Voltage readings for CPU0 8pin power connector:
    107 Same display for ipp006 24 pin and 8 pin cpu0
    108 
    109 Voltage readings for CPU0 8pin power connector:
    110 Same for CPU1 8pin.  pg 430ms
    111 
    112 
    113 
    114 == May 16 ==
    115 replace a raid card in ipp005
    116 
    117 
    118 == July 15 ==
    119 Swapped the 1st 3 mem sticks out of ippdb00
    120 
    121 == July 27 ==
    122 Spent AM diagnosing the non-boot problem in ipp030 - failed MB
    123 After lunch 
    124 Took failed MB out of ipp030 and replaced it with MB from ippc10.
    125 ippc10 remains out of the rack.
    126 
    127 == July 29 ==
    128 Added new memory to ippb00 1nd 01 - now have 32GB each.
    129 
    130 == Aug 1 ==
    131 Swapped all memory out of ippdb03 - formally ippc00 - using the memory from ippc10  - waiting for MB
    132 Added 2 2TB disks to ippc10 to become the newest rendition of ippdb03
    133 
    134  
    135 == Aug 15 ==
    136 ipp011
    137 Upgraded to new MB and 2 TB drives. Had to move CPU0 memory stick from outer blue slot to middle black slot.
    138 Suppose to fill blue slots first. The memory was not detected in the outer black slot. Same weirdness seen in ipp009 ( according to Gavin ASA chassis)
    139 
    140 ipp019
    141 Upgraded to new MB and 2 TB drives. Memory all in blue slots and detected
    142 *The 1st new MB installed did not detect any memory in the cpu0 no matter where we put the sticks. Tried a second new MB using all the same memory and CPU's - with good results.  Will RMA the 1st new MB.
    143 
    144 ipp015
    145 Upgraded to 2TB drives.  On reboot saw only 12GB of memory.  For this machine memory is suppose to be an all black slots.  However we noted that the memory was in a mixture of black and blue slots.  Moved to all blacks. All memory detected.
    146 
    147 == Aug 16 ==
    148 Installed the replacement MOBO for ippc10
    149 
    150 == Aug 23 2011 ==
    151 
    152 Haydn Bill and I were able upgrade ipp004,ipp013 and ipp018 MB and disks without incident.
    153 
    154 We corrected the existing error in the RAID card cabling in ipp004 so now the drives are seated in the same order as the other machines ( funnily we discovered and remembered this problem because we had missed swapping a drive and put a 500GB back in which was showing in the wrong slot )
    155 
    156 ipp037 has a new MB.
    157 
    158 Gavin installed the OS and building the RAID - he was instrumental once again in supporting us while at MHPCC.
    159 
    160 Haydn did all the drive replacements
    161 
    162 
    163 == Aug 30 ==
    164  Replaced IPP014's raid controller.
    165 
    166 
    167 == Aug 31 ==
    168  
    169 ipp010, ipp017, ipp020 (disk upgrades) and ipp021 (mobo
    170 upgrade).
    171 
    172 
    173 == Sept 13 ==
    174 
    175 
    176 Bill and Haydn Installed the new MOBO on ipp021
    177 
    178 
    179 
    180 == Sept 20 ==
    181 
    182 
    183 Bill and Haydn Installed the new MOBO on ipp020
    184 
    185 
    186 Sept 30
    187 
    188 installed 3TB disks in slots 2&3
    189 
    190 == Oct 10 ==
    191 
    192 Swapped CPU0 memory with CPU1 memory
    193 
    194 == OCT 19 ==
    195  * replaced ippb00 A0 drive 3 but still not being detected. worked with LSI to de-bug - needs hardware fix Ordered parts from YMI
    196 
    197 
    198 == OCT 24 ==
    199  * ipp036 has dead CPU0 fan  unable to boot.
    200  * Swapped MOBO out of ippc10 into ipp036 (still in rack - unable to remove it - only me working ) Left ipp036 with 1 processor - not sure if cpu damaged - did want to hurt MOBO of running system
    201  * Installed 6 of the 8 3TB drives in STARE nodes. 2 drive DOA sent for RMA
    202 
    203 == OCT 25 ==
    204  * ippb00 replaced cable,raid card and sled - still not detecting drive A0 3 - ordered back plane
    205 
    206 == OCT 27 ==
    207  * Installed spare ASUS MOBO into ippc10 - cpu heat sinks do not install properly - Brought ippc10 back to ATRC -ordered proper CPU carriage frame from YMI
    208 
    209 == NOV 1 ==
    210  * Installed ASUS MOBO into ippc10
    211 
    212 == Nov 4 ==
    213  * ippb00 Haydn replaced back plane for ippb00 a0 - Drive 3 detected. All drives put back in unit - parts sent back to YMI.
    214 
    215 == NOV 7 ==
    216  * team racking and installation of 13 5 U's
    217  * put cpu back into ipp036
    218  * put ippc10 back in rack
    219  
     174== Jan 21 ==
     175replaced motherboard hardware and disks for ipp009, ipp012, ipp014.
     176
     177ipp012 new disk 11 had to be swapped out.
     178ipp009: 6 4BG sticks were installed but only 5 are showing.  All the memory has been swapped out for new memory with the same result.
     179ipp014: We hear continuous beeps on power on changing too 2 short
     180beeps and a long one after something appears on the screen.  However
     181the 2 beeps and a long on are not constant - they become random.
     182We don't see any memory report on the first screen like the others.
     183We changed the mother board and memory with identical results.
     184There was a dead fan issue but we rewired it and all the fans are
     185working and the fan light is off.  No effect on the beep pattern. The
     186initial quick short beeps sound like memory beeps but it weird that we
     187get it to boot at all. Left out of the rack.
     188
     189The new mother boards do not have an IDE connection
     190for the existing optical drives.  We (Bill) drove to the only 2
     191computer stores on the island and purchased the only 2 notebook size
     192optical drives with sata connectors available. However - once we
     193installed these we realized the connectors were something we had not
     194seen - and did not have the cables for. At first glance they look like
     195sata connections but there is a power component as well.  opps.
     196Fortunately Brad had external usb optical drives we could use.
     197Hundreds of them, in fact.  Gavin had to Mac Gyvered them in place
     198because the cables were too short to set them anywhere.
     199
    220200
    221201= 2010 Production Node Work =