HOWTO: Run a Fedora 32-bit ARM VM emulated on x86-64

Background

I am writing this as a memory aid to myself the next time I have to do this, but this might make a nice addition to the Fedora ARM wiki once cleaned up.

I need a 32-bit ARM VM to debug some issues with armhfp image creation; the most efficient way to do this would be to use KVM on an aarch64 host like a Raspberry Pi 4, but mine is still in storage – and WFH means I’m getting too used to programming from my laptop while wandering around the house anyway. So … until my Pinebook Pro arrives, armhfp-on-x86_64 it is.

Initial installation

Per pwhalen on the ever-helpful #fedora-arm IRC channel on Freenode, UEFI installation is not supported yet for 32-bit ARM, and so the otherwise very relevant blog post by Marcin Juszkiewicz, Running VMs on Fedora/AArch64¹, does not apply here. Do not have edk2-arm installed, otherwise if you create a new ARM VM from virt-manager it will default to using UEFI mode.

Instead, mostly follow the Fedora QA test case². I’m replicating the instructions here in case something happens to the Fedora wiki:

• Download the latest arm disk image to $HOME:https://dl.fedoraproject.org/pub/fedora/linux/releases/32/Spins/armhfp/images/Fedora-Minimal-armhfp-32-1.6-sda.raw.xz
• Extract the disk image: unxz Fedora-Minimal-armhfp-32-1.6-sda.raw.xz
• Install virt-builder: sudo dnf install libguestfs-tools-c
• Extract the kernel/initrd from the disk image: virt-get-kernel -a Fedora-Minimal-armhfp-32-1.6-sda.raw
• Get kernel args for the the image: virt-cat -a Fedora-Minimal-armhfp-32-1.6-sda.raw /etc/extlinux.conf
• Copy the media to the the default libvirt image location: sudo mv Fedora-Minimal-armhfp-32-1.6-sda.raw vmlinuz-*.armv7hl initramfs-*.armv7hl.img /var/lib/libvirt/images/
• Ensure qemu-system-arm is installed: sudo dnf install qemu-system-arm

There is a typo in the Create the VM with virt-install section - the OS variant should be Fedora 32, not 31. Also I renamed the VM name from the example given, which uses the filename of the disk image, to the shorter fedora-armhfp-32.

❯ sudo virt-install \
    --name fedora-armhfp-32 --ram 4096 --arch armv7l --machine virt-2.11 --os-variant fedora32 --import \
    --disk /var/lib/libvirt/images/fedora-armhfp-32.raw \
    --boot kernel=/var/lib/libvirt/images/vmlinuz-5.6.6-300.fc32.armv7hl,initrd=/var/lib/libvirt/images/initramfs-5.6.6-300.fc32.armv7hl.img,kernel_args="console=ttyAMA0 rw root=LABEL=_/ rootwait"

I did not want to change too many things at the beginning, but if I were to do this again I’d convert the disk image to qcow2 before invoking virt-install so I don’t have to change the VM configuration. Not only is it more space efficient but you get to take snapshots – and when the emulation is really slow, you definitely want quick recovery from any mistake.

❯ sudo qemu-img convert -f raw -O qcow2 fedora-armhfp-32.{raw,qcow2}

Resizing

The default root partition was overly small – out of the box it already consumed 1.1G out of 1.3G! You definitely want to resize it ASAP.

RHEL6’s Virtualization Administration Guide³ comes to the rescue here; while dated the information is still very relevant.

[root@michel-fedora-PC198L6J images]# qemu-img resize fedora-armhfp-32.qcow2 20G
[root@michel-fedora-PC198L6J images]# ls
fedora-armhfp-32.qcow2  fedora-armhfp-32.raw  fedora.qcow2 Fedora-Workstation-armhfp-33-sda.raw  initramfs-5.6.6-300.fc32.armv7hl.img  vmlinuz-5.6.6-300.fc32.armv7hl
[root@michel-fedora-PC198L6J images]# cp -p fedora-armhfp-32.qcow2{,.backup}
[root@michel-fedora-PC198L6J images]# virt-df -h ./fedora-armhfp-32.qcow2
Filesystem                                Size       Used  Available  Use%
fedora-armhfp-32.qcow2:/dev/sda1           76M        25M        51M   34%
fedora-armhfp-32.qcow2:/dev/sda2          457M        89M       353M   20%
fedora-armhfp-32.qcow2:/dev/sda3          1.3G       1.1G       100M   90%
[root@michel-fedora-PC198L6J images]# virt-resize ./fedora-armhfp-32.qcow2.backup ./fedora-armhfp-32.qcow2 --expand /dev/sda3
[   0.0] Examining ./fedora-armhfp-32.qcow2.backup
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be resized from 1.3G to 19.4G.  The
filesystem ext4 on /dev/sda3 will be expanded using the ‘resize2fs’
method.

**********
[   2.3] Setting up initial partition table on ./fedora-armhfp-32.qcow2
[   3.7] Copying /dev/sda1
[   3.9] Copying /dev/sda2
[   4.5] Copying /dev/sda3
[   7.1] Expanding /dev/sda3 using the ‘resize2fs’ method

Resize operation completed with no errors.  Before deleting the old disk,
carefully check that the resized disk boots and works correctly.

Kernel updates

Upgrading via dnf upgrade works as usual. However, because we hardcode the kernel and initrd information in the VM configuration, switching to boot the new kernel requires some manual changes: use virt-get-kernel, as done initially to extract the kernel and initrd out of the original disk image; this will now extract the newer versions.

lib/libvirt/images 
❯ sudo ls -l                          
[sudo] password for michel: 
total 84002632
-rw-r--r--. 1 qemu   qemu    5895618560 Aug 20 18:22 fedora-armhfp-32.qcow2
-rw-r--r--. 1 qemu   qemu    1436942656 Aug 20 15:59 fedora-armhfp-32.qcow2.backup
-rw-rw-r--. 1 qemu   qemu    2088763392 Aug 20 15:57 fedora-armhfp-32.raw
-rw-------. 1 qemu   qemu   71195557888 Aug  8 14:25 fedora.qcow2
-rw-r--r--. 1 root   root    7121928192 Aug 20 12:53 Fedora-Workstation-armhfp-33-sda.raw
-rw-rw-r--. 1 michel michel    51619027 Aug 20 14:39 initramfs-5.6.6-300.fc32.armv7hl.img
-rw-rw-r--. 1 michel michel     7643648 Aug 20 14:39 vmlinuz-5.6.6-300.fc32.armv7hl

lib/libvirt/images took 3s 
❯ sudo virt-get-kernel -a fedora-armhfp-32.qcow2
download: /boot/vmlinuz-5.7.15-200.fc32.armv7hl -> ./vmlinuz-5.7.15-200.fc32.armv7hl
download: /boot/initramfs-5.7.15-200.fc32.armv7hl.img -> ./initramfs-5.7.15-200.fc32.armv7hl.img

lib/libvirt/images took 5s 
❯ sudo ls -l                                    
total 84028332
-rw-r--r--. 1 qemu   qemu    5895618560 Aug 20 18:22 fedora-armhfp-32.qcow2
-rw-r--r--. 1 qemu   qemu    1436942656 Aug 20 15:59 fedora-armhfp-32.qcow2.backup
-rw-rw-r--. 1 qemu   qemu    2088763392 Aug 20 15:57 fedora-armhfp-32.raw
-rw-------. 1 qemu   qemu   71195557888 Aug  8 14:25 fedora.qcow2
-rw-r--r--. 1 root   root    7121928192 Aug 20 12:53 Fedora-Workstation-armhfp-33-sda.raw
-rw-rw-r--. 1 michel michel    51619027 Aug 20 14:39 initramfs-5.6.6-300.fc32.armv7hl.img
-rw-r--r--. 1 root   root      18473488 Aug 20 18:23 initramfs-5.7.15-200.fc32.armv7hl.img
-rw-rw-r--. 1 michel michel     7643648 Aug 20 14:39 vmlinuz-5.6.6-300.fc32.armv7hl
-rw-r--r--. 1 root   root       7836160 Aug 20 18:23 vmlinuz-5.7.15-200.fc32.armv7hl

Now we just need to switch the VM configuration to use the new kernel. The configuration files are in /etc/libvirt/qemu:

❯ sudo find /etc/libvirt/qemu -maxdepth 1 -name '*.xml'
/etc/libvirt/qemu/fedora-armhfp-32.xml
/etc/libvirt/qemu/fedora.xml

Either edit the file directly, or from virt-manager, open the VM and select OS information. You must have XML editing enabled though (in Virtual Machine Manager, change this in Edit -> Preferences).

Unfortunately, since the kernel and initrd still have to be specified in the XML configuration, that means they are not part of any snapshot you might take.

Conclusion

There we are! This is definitely not the fastest way to run 32-bit ARM – on my ThinkPad T490s booting the emulated VMs take minutes, and the initial dnf upgrade probably took over an hour to upgrade and verify a minimal set of ~ 200 packages. But it does get the job done, and apart from the slowness, it behaves just like any other libvirt VM, which is nice.

This post is day 3 of my #100DaysToOffload challenge. Visit https://100daystooffload.com to get more info, or to get involved.