arch linux arm aarch64 + ovmf uefi + qemu

since it’s some time i’m dealing with this stuff, i decided it would be a good idea to put everything on here for further reference. following are the steps to build an arch linux armv8 aarch64 qemu image, with efi and efistub boot.

pre assumptions:

everything is done on arch linux. it should be pretty distro-intependent though
qemu and qemu-arch-extra (qemu-system-aarch64) packages installed
ovmf-aarch64 package installed
all commands as root

can take this guide as initial reference, especially for stesps 1 to 7: juno archlinuxarm guide:

so first we create a disk file:

↬  dd if=/dev/zero of=arm64disk.img bs=100M count=80
8283750400 bytes (8.3 GB, 7.7 GiB) copied, 48 s, 172 MB/s
80+0 records in
80+0 records out
8388608000 bytes (8.4 GB, 7.8 GiB) copied, 48.7326 s, 172 MB/s

make efi and root partitions:

↬  parted arm64disk.img
GNU Parted 3.2
Using /home/andrei/Documents/armarchguide/arm64disk.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
(parted) mkpart ESP fat32 1MiB 513MiB                                    
(parted) set 1 boot on                                                    
(parted) mkpart primary ext4 513MiB 100%                                  
(parted) quit

create loop device so we can mount the partitions:

↬  losetup -fP arm64disk.img
↬  losetup -a
/dev/loop0: [2050]:8134417 (/home/andrei/Documents/armarchguide/arm64disk.img)
↬  fdisk -l /dev/loop0
Disk /dev/loop0: 7.83 GiB, 8388608000 bytes, 16384000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7B57E083-2820-4C1B-87DA-E0E9BDDD092D

Device         Start      End  Sectors  Size Type
/dev/loop0p1    2048  1050623  1048576  512M EFI System
/dev/loop0p2 1050624 16381951 15331328  7.3G Linux filesystem

create the file systems; fat32 for efi system partition and ext4 for root:

↬  mkfs.vfat -F 32 -n "arch arm efi" /dev/loop0p1
mkfs.fat 4.1 (2017-01-24)
mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
↬  mkfs.ext4 /dev/loop0p2
mke2fs 1.45.4 (23-Sep-2019)
Discarding device blocks: done                            
Creating filesystem with 1916416 4k blocks and 479552 inodes
Filesystem UUID: bbcea09a-9fbb-426a-85ea-cae36d3e082f
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

mount and copy over the root filesystem provided by arch linux arm:

↬  cd /mnt
↬  mkdir armroot
↬  mkdir armboot
↬  mount /dev/loop0p1 armboot/
↬  mount /dev/loop0p2 armroot/
↬  wget http://os.archlinuxarm.org/os/ArchLinuxARM-aarch64-latest.tar.gz
↬  bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C armroot/
↬  sync
↬  mv armroot/boot/* armboot/

write something like this on the guest’s /etc/fstab; this is to instruct the system to correctly mount the partition at the right place when booting. if you get into some recovery mode or in general the guest doesn’t boot, this is one of the first places you should look.

↬  cat armroot/etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
/dev/vda1      	/boot     	vfat      	defaults,rw	0 2
/dev/vda2	/         	ext4      	rw,relatime	0 1

since there’s no boot manager nor any boot process set up in the guest system, we’ll need the Image and vmlinuz-linux from the boot partitions for the first boot, so copy them over. this way we can start the kernel externally, specifying the root partition to use and bootstrap from there.

↬  cp /mnt/armboot/Image .
↬  cp /mnt/armboot/initramfs-linux.img .

we can now unmount the partitions.

↬  umount armroot
↬  umount armboot

using the two files copied before, we can boot the guest. login credentials are root:root:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a57 -kernel ./Image -initrd ./initramfs-linux.img -drive file=arm64disk.img,format=raw,index=0,media=disk  -nographic -no-reboot  -append "root=/dev/vda2 rw console=ttyAMA0" -net nic -net bridge,br=qemu_wifibr

side note: you can use port forwarding to enable networking on the guest specifying the -net user,hostfwd=tcp::5022-:22 option instead of -net bridge,br=qemu_wifibr. read the appendix for more info

if you find you can ping 8.8.8.8 but not google.com, you just have to set up some nameserver:

[root@alarm ~]# echo 'nameserver 8.8.8.8' > /etc/resolv.conf

first thing i usually do is upgrade the system. a couple of problems arise though:

first is pacman was hanging, i commented the ‘CheckSpace’ line in /etc/pacman.conf as suggested here and for some reason it solved the issue.

second is the keyring;

[root@alarm ~]# pacman -Syu
[..]
(55/55) checking keys in keyring                   [######################] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
downloading required keys...
error: key "77193F152BDBE6A6" could not be looked up remotely
error: required key missing from keyring
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.

easily solved by initializing it:

[root@alarm ~]# pacman-key --init
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key EDDF1D2FC6017A2C marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/EF181BC21A79C65FBD4EF10DEDDF1D2FC6017A2C.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
[root@alarm ~]# pacman-key --populate archlinuxarm
==> Appending keys from archlinuxarm.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key 69DD6C8FD314223E14362848BF7EEF7A9C6B5765...
  -> Locally signing key 02922214DE8981D14DC2ACABBC704E86B823CD25...
  -> Locally signing key 9D22B7BB678DC056B1F7723CB55C5315DCD9EE1A...
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   3  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   3  signed:   1  trust: 0-, 0q, 0n, 3m, 0f, 0u
gpg: depth: 2  valid:   1  signed:   0  trust: 1-, 0q, 0n, 0m, 0f, 0u

now it upgrades successfully. install some useful package (vim, binutils) and efibootmgr; then poweroff to boot with the efi variables.

the aarch64 efi firmware is found here:

↬  pacman -Ql ovmf-aarch64
ovmf-aarch64 /usr/
ovmf-aarch64 /usr/share/
ovmf-aarch64 /usr/share/ovmf/
ovmf-aarch64 /usr/share/ovmf/AARCH64/
ovmf-aarch64 /usr/share/ovmf/AARCH64/QEMU_EFI.fd

create efivars file:

↬  dd if=/dev/zero of=arm64efivars.img bs=128K count=1
1+0 records in
1+0 records out
131072 bytes (131 kB, 128 KiB) copied, 0.010478 s, 12.5 MB/s

now, trying to boot with them prints this weird error:

↬  qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a57 -kernel ./Image -initrd ./initramfs-linux.img -drive file=arm64disk.img,format=raw,index=0,media=disk  -nographic -no-reboot  -append "root=/dev/vda2 rw console=ttyAMA0" -net nic -net bridge,br=qemu_wifibr -drive file=/usr/share/ovmf/AARCH64/QEMU_EFI.fd,if=pflash,format=raw,readonly -drive file=./arm64efivars.img,if=pflash,format=raw
qemu-system-aarch64: Initialization of device cfi.pflash01 failed: device requires 67108864 bytes, block backend provides 2097152 bytes

solution: trick it into thinking it’s 64MB (or just really pad the files to 64MB).

↬  cp /usr/share/ovmf/AARCH64/QEMU_EFI.fd arm64eficode.fd
↬  dd if=/dev/zero of=arm64eficode.fd bs=1c count=1 seek=67108863
↬  'du' arm64eficode.fd
2052	arm64eficode.fd
↬  'ls' -l arm64eficode.fd
-rw-r--r-- 1 root root 67108864 Oct  3 22:51 arm64eficode.fd

same thing for arm64efivars.fs.

now workz:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a57 -kernel ./Image -initrd ./initramfs-linux.img -drive file=arm64disk.img,format=raw,index=0,media=disk  -nographic -no-reboot  -append "root=/dev/vda2 rw console=ttyAMA0" -net nic -net bridge,br=qemu_wifibr -drive file=arm64eficode.fd,if=pflash,format=raw,readonly -drive file=./arm64efivars.fd,if=pflash,format=raw

we have efivars!

[root@alarm ~]# efibootmgr -v
Timeout: 3 seconds
BootOrder: 0000
Boot0000* UiApp	FvVol(64074afe-340a-4be6-94ba-91b5b4d0f71e)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)

let’s add the efi boot stub:

[root@alarm ~]# blkid
/dev/vda1: LABEL_FATBOOT="arch arm ef" LABEL="arch arm ef" UUID="FF4C-92F7" TYPE="vfat" PARTLABEL="ESP" PARTUUID="78e259bb-c29e-4afd-94d8-30b0ff5bad82"
/dev/vda2: UUID="bbcea09a-9fbb-426a-85ea-cae36d3e082f" TYPE="ext4" PARTLABEL="primary" PARTUUID="12a403e4-72ee-49a5-8c59-9b4c4475a4c8"
[root@alarm ~]# efibootmgr --disk /dev/vda --part 1 --create --label "Arch Linux ARM" --loader /Image --unicode 'root=PARTUUID=12a403e4-72ee-49a5-8c59-9b4c4475a4c8 rw initrd=/initramfs-linux.img' --verbose

and without kernel and initrd, boots ok:

qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a57 -drive file=arm64disk.img,format=raw,index=0,media=disk -nographic -no-reboot -net nic -net bridge,br=qemu_wifibr -drive file=arm64eficode.fd,if=pflash,format=raw,readonly -drive file=./arm64efivars.fd,if=pflash,format=raw

appendix - networking

the bridge is a better solution since the guest appears to be just another host in the network. you can easly create one to share your ethernet (wired) connection; wifi sharing is not so easy though. i use parprouted, creating an arp-proxy, and the following script to automate its creation and deletion:

#!/usr/bin/env bash

printusage() {
		echo "usage: $0 start/stop"
		exit 1
}

if [[ "$EUID" -ne 0 ]]; then
	echo "need root powa!"
	exit 1
fi

if [[ -z "$1" ]]; then
	printusage
fi

action="$1"

br_if="qemu_wifibr"
wifi_if="$(iw dev | awk '$1=="Interface"{print $2}' | head -n 1)"
parprouted_if_addr="10.11.12.1/32"

if ! grep -q "allow ""$br_if" /etc/qemu/bridge.conf; then
	echo "allow ""$br_if" >> /etc/qemu/bridge.conf
fi

case "$action" in
	start)
		brctl addbr "$br_if"
		ip addr add "$parprouted_if_addr" dev "$br_if"
		ip link set "$br_if" up
		parprouted "$wifi_if" "$br_if"
		echo "$br_if created"
		;;
	stop)
		ip addr del "$parprouted_if_addr" dev "$br_if"
		ip link set dev "$br_if" down
		brctl delbr "$br_if"
		echo "$br_if deleted"
		;;
	*)
		printusage
		;;
esac

keep in mind that dhcp and other lower network level stuff might not work; so for example you’d have to set addresses manually in the guest…

[root@alarm ~]# ip ad add 192.168.1.201/24 dev enp0s1
[root@alarm ~]# route add default gw 192.168.1.1
[root@alarm ~]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=986 ms