LSI 9211-8i

2019-03-30 22:43

On Friday I got the new/old SAS controller from China. As expected, it had old firmware and it was the IR version (RAID). As whole point of this new controller was to get access to my tape drive and not have to deal with hardware RAID, what I really wanted was the IT version, also known as HBA mode.

Having updated the firmware in the built-in controller in my server I already had a bootable disk with firmwares courtesy of Broadcom né LSI etc., including the firmware for the 9211. Or so I thought (see below).

Firmware Adventure

I started by mounting the card in my game box and booting the usb stick with the firmware updates. This didn't really work; the software wouldn't see the card and I guessed that it was because this system--as opposed to the supermicro --uses UEFI.

Googling the problem, I found some guides and figured out that there is a efi program version of the firmware flash utility (sas2flash.efi). This file is helpfully provided by Broadcom, they didn't have a page for the 9211, but I found the utility on the 9210 downloads page I'm not quite sure but as far as I've been able to make out, the 9210, 9211 and 9240 and a few others are in fact the same card but with different firmware (some might have more ports than others). LSI also OEMd the card to a lot of server manufacturers (including IBM) and they're supposedly mostly cross-flashable. It wasn't interesting to me so I didn't dive too deep into that.

Having acquired the flash utility, I tried booting the usb stick in EFI mode, but that didn't work, it wasn't set up for it and my experience with EFI is extremely limited, not having fiddled with it much. It was worth a shot though.

More googling taught me that I would need an UEFI Shell to run the flash utility, that my Asus BIOS doesn't have one built in, where to find a development UEFI Shell and how to set up the usb stick to boot it. Pretty easy actually.

Make an UEFI bootable USB device

  1. Make sure that your usb stick is FAT32
  2. Create \EFI\Boot\
  3. Stick your bootloader/shell in the above directory with the name bootx64.efi for 64-bit UEFI. If your UEFI is 32-bit, you probably need to have a 32-bit version of the bootloader/shell and call it bootia32.efi.

From helpful guides from people crossflashing I found a uefi shell utility on the Tianocore EDK II github page. Turns out that you actually need an old version of the shell, the latest one refused to run the flash utility.

Moving on...

Confident in that I had everything I needed; a SAS ROM image from the 9210 downloads and the 9211it.img file, I booted into the UEFI Shell, found the usb stick by running map and mounted it using the mount command.

I then ran sas2flash.efi -o -e 6, the magical command that according to the crossflashing guides is required to switch from IR to IT firmware. It found my controller and ran the erased the PROM. The guides also warned that if you reboot your system with the card in this state, it will be bricked.

So I move on to flashing the new firmwares and... "INVALID FIRMWARE IMAGE".

*gulp*

But the image file I had was an official firmware image, perhaps it needed to be extracted somehow? I quickly fired up my NetBSD laptop, yanked the USB stick out of the PC and used file to look at the img file. "DOS/MBR boot sector". A-ha! I mounted the filesystem using a vnd(4) device and found two interesting files, 2114it.bin and 2118it.bin... This must be the ROMs for the 4i and 8i variants of the card! I copied the files back to the USB stick, stuck it back in the PC and now sas2flash.efi ran without issue. Phew!

I then rebooted the system, crossing my fingers and voila, firmware v20 with MPTSAS ROM 2.0.

Server surgery

Saturday morning arrives (alright, noon) and it was time for surgery. I shut down my server and pulled it out of the rack (woho, rails). I began by pulling all the SAS cables and the expander, since I'd be rearranging everything anyway.

I wired it up as I planned in the last blog post, and except for actually managing to get two cables wrong which resulted in a parity rebuild on one of the arrays on the SMC2108, it went without a hitch.

Upon booting, NetBSD found the controller:

mpii0 at pci4 dev 0 function 0: vendor 1000 product 0072 (rev. 0x03)
mpii0: interrupting at ioapic1 pin 6, event channel 7
mpii0: SGX-SAS6-INT-Z, firmware 20.0.4.0, MPI 2.0

All the disks;

mpii0: physical disk inserted in slot 1
mpii0: physical disk inserted in slot 2
mpii0: physical disk inserted in slot 3

[...]

sd2 at scsibus1 target 1 lun 0: <HP, EG0146FAWHU, HPDF> disk fixed
sd2: 136 GB, 114029 cyl, 2 head, 1257 sec, 512 bytes/sect x 286749488 sectors
sd2: tagged queueing
sd3 at scsibus1 target 2 lun 0: <HP, EG0146FAWHU, HPDF> disk fixed
sd3: 136 GB, 114029 cyl, 2 head, 1257 sec, 512 bytes/sect x 286749488 sectors
sd3: tagged queueing
sd4 at scsibus1 target 3 lun 0: <HP, EG0146FAWHU, HPDF> disk fixed
sd4: 136 GB, 114029 cyl, 2 head, 1257 sec, 512 bytes/sect x 286749488 sectors
sd4: tagged queueing

And to my surprise, the Expander:

mpii0: physical disk inserted in slot 36

[...]

ses0 at scsibus1 target 36 lun 0: <HP, HP SAS EXP Card, 2.08> enclosure services fixed
ses0: SCSI-3 SES Device
ses0: tagged queueing

Awesome.

It didn't immediately see the tape drive, because the external enclosure wasn't turned on, but as soon as I did, the controller noticed the tape drive

mpii0: physical disk inserted in slot 24

It was then a matter of scanning the bus scsictl scsibus1 scan 24 0 to attach the tape drive;

st0 at scsibus1 target 24 lun 0: <QUANTUM, ULTRIUM 5, 3180> tape removable
st0: drive empty
st0: tagged queueing

And that was it, adventure concluded. Almost.

Broken disks?

As I now had three old 147GB drives in the system, I tried building them into a software RAID 5 set using the NetBSD raidframe system, but that turned out not to work, I started getting non-media errors from the disks.

sd3(mpii0:0:2:0): non-media hardware failure, info = 1513 (decimal), data = 00 00 00 00 32 00 9d 80 00 96
sd3a: error reading fsbn 1472 of 1472-1535 (sd3 bn 1472; cn 0 tn 1 sn 215)

This seemed a bit worrying, but I knew from experience that at least one of those drives had had some failure when they were in the HP machines I tried using before I got the SuperMicro. I wasn't certain though, since this was a non-media error, it could be the controller, cable or backplane as well.

I tried moving them to different slots in the backplane (ones that are on a different SAS port) but that didn't seem to help. They just got more errors.

I ranted a bit on IRC about it, and was reminded of SMART, so I checked the smart status of the disks and sure enough, all three of them had IMMINENT FAILURE warnings. I guess that's where I'll leave it for now, those drives weren't ones that I would be using anyway, being so small and having 45k operating hours each.

It was a fun set of adventures anyway, SAS really appears to be a nice system, things mostly Just Work(tm).

Now it's just wating for my order of LTO-5 tapes to arrive and I can get back to backing up stuff.

Staffan