Thursday, May 14, 2009

What does a toaster and a HP DL360g5 have in common?

Why both can melt plastic very well.

Today one of the SysAdmins came to me with the woeful tale of not being able to PXE boot a DL360 so he could load an OS on it. After using iLO to look at the system remotely we see that the system never sees any drives. So off we go on a short walk to the data center and see what is going on. It should be noted that iLO gave no health warnings. It was all peachy keen save hard drives not showing up.

We arrive in the data center, plug in the monitor and see that yes the drives are not showing up.  The lack of blinking lights on the drives should have been our first clue but we skipped that step as it is rarely that two out of two drives fail. But we were obviosly wrong.

First I pulled out the drive on bay one and all was well. Then I pulled out drive number 2.



Well that is a sure sign of a problem. So I pulled the DL360g5 from the rack and took it to my desk. Fortunately we had another server on hand so the Admin was able to get back to work rebuilding his system. I took the system back to my desk and started to crack it open so as to see what other damage there might be.

Here is a view of the burned out fan.


And the burned drive controller.



It was fun talking to the support people and explaining that "no it was not in a fire or struck by lightning. It caused a fire or electrical arc". The engineer that will be coming out to document it kind of laughed when his boss told him they actually have a procedure for this but it is rarely used.

To be fair to HP this is the first I've ever heard of a server actually starting the fire. We have around 100 DL360's in service and this is the only one that we have had this happen to. I like the DL360 line and wish I could get the DL360g6 as they have are using the new Intel® Xeon® processor 5500 and have a huge energy savings. However I will be interested to see what the HP engineer says when he comes to look at the system.

UPDATE:

I have some more photos for when we moved the fans out of the way.

First up some melted fans.

And here is a series of shots for the drive controller.







Cannot wait to hear back from the HP Engineers to see what the failure was.