Our thinking

Recover a Ubiquiti EdgeRouter with a failed USB Flash, or ERMK Redux

In the past, I’ve had a number of the original Ubiquiti EdgeRouter Lite units fail – they were all the original plastic-case units, and in each case it was the USB flash that died. It was difficult to pinpoint exactly what caused the failure as the entire system runs from RAM, so it remains up and keeps forwarding packets even if the flash has died. It’s only when you go to reboot that you suddenly notice that there is an error.

This all seemed to be fixed with the second-generation units in a metal case. The only failures I’ve seen with these units in the field were failed power supplies.

Until now. I went to see a client who had a failed router. I suspected the power supply had failed, and indeed it was dead. Upon replacing the power supply however, it was quickly apparent that the flash had died as well.

There used to be an EMRK – EdgeMax Rescue Kit that was a collaborative effort between some Ubiquity forum members, and with a bit of input from Ubiquiti as well.

EdgeMax rescue kit (now you can reinstall EdgeOS from scratch)

This method was always rather complicated to set up as you needed a particular firmware image to boot and you needed to set up a tftp server to load the firmware over the network.

There is now a newer method, called mkeosfs which are some scripts on GitHub. You still need a Linux instance to create the ext3 filesystem on the USB, but you don’t need to do anything over the network.

I tried this, and it didn’t work for me, however I now suspect that the first USB drive I was using had problems. I also found some pre-built images, however the 2GB image was slightly bigger than the 2GB flash drive I was using, so the filesystem was broken when I dd’d it to the USB.

I also managed to track down the steps to create a bootable USB manually, which I’ll copy here for future reference if I ever need it again.

In the EMRK forum thread, there was a post from BranoB on how to re-create the USB drive manually.

It’s quite straightforward – download the latest firmware image from Ubiquiti, create two partitions on the USB, initialise the filesystem on each and then copy the relevant files to each partition.

Steps follow:

Download the firmware from UBNT download page and follow steps below (I’ll use Linux examples but this can be done on Windows and OS-X as well given proper tools and little expertise).

Insert new USB drive into the PC and create MSDOS partition table with two partitions. 1st partition FAT32 150MB and 2nd partition is ext3 with size to end of the drive (use min. 2GB drive)

(my new USB drive in this example is /dev/sdc)

# parted -s /dev/sdc mktable msdos
# parted -s /dev/sdc mkpart primary fat32 1 150MB
# parted -s /dev/sdc mkpart primary ext3 150MB 100%
# mkfs.vfat /dev/sdc1
# mkfs.ext3 -q /dev/sdc2

Download the firmware and untar it

# wget http://dl.ubnt.com/firmwares/edgemax/v1.8.0/ER-e100.v1.8.0.4853089.tar
# tar xf ER-e100.v1.8.0.4853089.tar

Create mount points and mount the partitions

# mkdir /tmp/er_sdc1 /tmp/er_sdc2
# mount /dev/sdc1 /tmp/er_sdc1
# mount /dev/sdc2 /tmp/er_sdc2

Copy the extracted firmware files to the respective locations (pay attention to filenames)

# cp vmlinux.tmp /tmp/er_sdc1/vmlinux.64
# cp vmlinux.tmp.md5 /tmp/er_sdc1/vmlinux.64.md5

# cp squashfs.tmp /tmp/er_sdc2/squashfs.img
# cp squashfs.tmp.md5 /tmp/er_sdc2/squashfs.img.md5
# cp version.tmp /tmp/er_sdc2/version
# mkdir -p /tmp/er_sdc2/w

# sync

At this point you’re done. Next step is optional to verify files integrity.

# diff -s <(md5sum /tmp/er_sdc1/vmlinux.64 | awk -F ' ' '{print $1}') <(cat /tmp/er_sdc1/vmlinux.64.md5)
Files /dev/fd/63 and /dev/fd/62 are identical

# diff -s <(md5sum /tmp/er_sdc2/squashfs.img | awk -F ' ' '{print $1}') <(cat /tmp/er_sdc2/squashfs.img.md5)
Files /dev/fd/63 and /dev/fd/62 are identical

Unmount the USB drive

# umount /tmp/er_sdc1 /tmp/er_sdc2

Done. Place the USB drive into ER and boot up.

Leave a Reply