MacOS on Ubuntu with KVM/QEMU and OVMF/Clover EFI

Some guidance on running macOS (up to Catalina) on KVM

MacOS on Ubuntu with KVM/QEMU and OVMF/Clover EFI

Status - 17 July 2020 - Core information collected [Optionally: I may at some point add section on moving to / from: Physical to Virtual Mac and to / from VMWare Fusion to QEMU / KVM Macs]. Now verified with Ubuntu 20.04 and completed some further Q35 Version and Network device testing.

If you do Mac development then its likely you will run virtual Mac's. In fact runing virtual machines for development is pretty much essential for anything that is non-trival. For Mac developer, this meant a choice of VMWare or Parallels and now you can also use Linux with KVM / QEMU and Clover/OVMF.

Here are the high level steps and instructions required to get macOS (OS X) up and running on Ubuntu with KVM / QEMU.

  1. Get Ubuntu running with virtualisation enabled
  2. Install KVM / QEMU software
  3. Configure Ubuntu network bridge with NetPlan
  4. Add Clover for EFI boot support
  5. Create your Mac VM using the right QEMU configuration and settings for macOS
  6. Optionally add PCI Pass-Through for network and GPU support on you virtualised macOS machine

I will cover each area of configuration in turn.

Ubuntu Install and Configuration

I started experimenting with macOS on KVM with Ubuntu 16.04. At the time this required recompiling of QEMU.  I then moved to Ubuntu 18.04 (which does not require any compilation) and updated to 19.04 (which has much much better performance) and have tested with 19.10. I am now running 20.04 LTS and have found it completely stable for running virtual macOS on KVM.

Here is summary of various Ubuntu Versions (desktop):

  • Ubuntu 16.04 - Requires compile of upstream QEMU (avoid)
  • Ubuntu 18.04 - Works out of box, but introduces NetPlan, so you need to learn how to configure your bridge (linux virtual switch) to get your networking working, also it is very very slow to boot macOS
  • Ubuntu 19.04 - As per 18.04 and boots macOS much much faster
  • Ubuntu 19.10 - having done testing, I am now using 19.10 as main virtualisation host
  • Ubuntu 20.04 - have tested and have running number of macOS vm's as part of upgrade process from 19.10

If you want to use PCI Pass-through the you should update your grub boot configuration, here are KVM Kernal Configuration parameters that are relevant:

 -- edit grub defaults and as per direction ensure you do sudo update-grub afterward

> cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'
GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 intel_iommu=on ipv6.disable=1"
-- NOTE: the iommu=1 intel_iommu=on are critical if you want to use PCI Pass-through

-- Next ensure nested virtualisation in on
> cat cat /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1

-- Tweek KVM
> cat /sys/module/kvm/parameters/ignore_msrs

NOTE: The "iommu=1 intel_iommu=on" grub settings are required to enable PCI Pass-Through

Reboot the machine to ensure all kernal parameter are set

Install QEMU / KVM Packages

The packages required for Ubuntu are: qemu, libvirt, virtual machine manager, OVMF:

sudo apt install qemu-system-x86 libvirt-clients libvirt-daemon-system ovmf virt-manager

This installs:

  • 64 bit x86 emulation and the libvirt abstraction layer,
  • the open virtual machine firmware (ovmf) - First level of EFI support (you need Clover as well)
  • virt-manager - virtual machine configuration and management UI

Configure NetPlan bridge

All networking (unless you attach a physical NIC via PCI Passthrough) with KVM / QEMU is handled via Linux Bridge. The Linux Bridge is a kernal loaded modue that behaves like a dumb ethernet switch. It does not provide any active network function in macOS context and the only important thing you have to decide is a name for the logical bridge (i.e. br20) and the physical NIC that this will be connected to. In Ubuntu 18.04, 19.04, 19.10 & 20.04 this is managed via the NetPlan configuration.

Here is a example NetPlan configuration for a machine with multiple ethernet NICs installed. You will see that all the network ethernet interfaces bar ens4f0 are IP disabled, Instead all the interfaces have bridges defined: br01, br20, br40 & br50. All the bridges have DHCP4 disabled. If you enable DHCP4 or define a static IP address on a bridge then this IP address will be assigned to the KVM Hosting machines and allow the virtual machines connected to the network bridge to get access to the hosting machine. This is likely to be undesireable for security reasons. It illustrates how virtualisation has signficant impact on how physical network seperation has to be managed. In this example the address is used to access the KVM hosting machine.

cat 01-network-manager-all.yaml 
# Let NetworkManager manage all devices on this system
  version: 2
  renderer: networkd
      dhcp4: no
      dhcp4: no
      dhcp4: no
      dhcp4: no
      dhcp4: no
      addresses: []
        search: []
        addresses: [XXX.XXX.XXX.4, XXX.XXX.XXX.5]
      interfaces: [ens2f0]
      dhcp4: no
        stp: false
        forward-delay: 0
      interfaces: [ens4f1]
      dhcp4: no
        stp: false
        forward-delay: 0
      interfaces: [eno1]
      dhcp4: no
        stp: false
        forward-delay: 0
      interfaces: [eno2]
      dhcp4: no
        stp: false
        forward-delay: 0

In my case the logical NICs that the ethernet ports connect to have corresponding seperate ethernet VLAN amd IP subnets. You need to keep in mind what subnet you want your macOS virtual machine to connect to and the bridge this is supported by. The selection of which bridge a NIC is connected to is managed via QEMU configuration.

Add Clover for MacOS EFI Support

While ovmf provides the first level of EFI boot for virtual Mac, it does not provide everything you need. The EFI boot process passes control to Clover EFI boot manager which provides support to load the HFS+ and new Apple File System (APFS) file system drivers needed to boot macOS (OS X).

Clover also provides things that make KVM / QEMU look like a real Mac to the macOS software layer.

To use Clover you need to create a small seperate qemu qcow2 virtual disk.  This contains the various Clover drivers and their configuration data. The Clover virtual disk becames the boot disk which then loads macOS from the HFS+/APFS virtual disk.

Setting up Clover boot disk is a bit of a chicken / egg problem, as the Clover intaller is a macOS app and the disk you want to do the install on is a virtual disk, but you need to have a bootable Mac to do this...

To start the process I created the Clover disk via VMWare Fusion macOS virtual machine, which had Clover disk attached as virtual image. So process is to:

  1. Install Clover onto the VMWare vmdk disk image
  2. Convert the vmdk format disk image to native QEMU qcow2 one (on Linux box: "qemu-image convert -f vmdk -O qcow2 <VM-DISK>.vmdk <QCOW-DISK>.qcow2")

Having Clover on a little disk makes it much easer to work with, than using a CD-ROM boot image.  The Clover boot disk is a MS FAT32 formated disk, which is setup based on EFI conventions.

Here is summary information on layout of Clover disk:

# diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *536.9 MB   disk0
   1:       Microsoft Basic Data EFI                     209.7 MB   disk0s1
   2:       Microsoft Basic Data CLOVER                  325.1 MB   disk0s2

The sizes correspond to: 512 MiB GPT Partitioned Disk with

  • 200 MiB MS FAT32 - EFI volume
  • 310 MiB MS FAT32 - CLOVER volume

Only the the EFI part is required, but having the extra CLOVER volumes is useful as you can put drivers and other things like "Clover Configurator" here as it may be that you cannot get access to network on initial boot.

To install clover on the disk you simply download Clover installer, run the installer and select the "/Volumes/EFI" (the mount point that partition will be on after rebooting with Clover disk attached) as the installation location. The installer will populate the Clover directory structure and add the files

Clover configuration example (without SMBIOS details):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">
        ... see below

On KVM the setup is very sensitive to the SMBIOS configuration and this is quite complex. The recommended approach is to use "Glover Configurator" to create this. Here is a seperate SMBIOS Clover configuration example:

        <string>Apple Inc.</string>
        <string>Apple Inc.</string>
        <string>Apple Inc.</string>
        <string>Mac mini</string>
        <string>Part Component</string>
        <string>Apple Inc.</string>

I have found the MacMini6,2 machine type to be reliable and have left my machine type as this for more than 1 year, as testing with other machines types resulted in boot failure.

You can test with Clover Configurator for more recent machines by loading and saving the configuration and then copying into backup a file on the EFI volume. This means you can readily return to a known configuration, by booting using Ubuntu and mounting the EFI volume. When editing the configuration, the important things is to ensure machine uuid and serial numbers are unique to the machine instance.

The Clover Configuration is with the EFI directory: /Volumes/EFI/EFI/CLOVER/config.plist

In addition to setting the Clover configuration, you also need to select a subset of EFI drivers to load on boot.

With Clover V2.5 Rel 5070 I have installed (from macOS terminal, on EFI boot disk mounted at /Volumes/EFI) the following drivers:

cd /Volumes/EFI/EFI/CLOVER/
mac:CLOVER admin$ ls
ACPI			clover-v2.5k_r5070	i-am-mac-catalina
CLOVERX64.efi		config.plist		kexts
OEM			doc			misc
ROM			drivers			themes
bak			drivers64UEFI		tools
mac:CLOVER admin$ cd drivers
mac:drivers admin$ ls
mac:drivers admin$ cd UEFI
mac:UEFI admin$ ls
AppleImageCodec.efi	DataHubDxe.efi		HashServiceFix.efi
AppleKeyAggregator.efi	FSInject.efi		SMCHelper.efi
AppleKeyFeeder.efi	FirmwareVolume.efi	apfs.efi
AppleUITheme.efi	HFSPlus.efi

To know what Clover system is used to boot from by convention I also do: "touch clover-vX.X_rxxxx"and "touch i-am-MAC-OS" in the /Volume/EFI/EFI/CLOVER directory so it is easy to see the Clover version and virtual Mac that is being used. This is very helpful if you have multiple Clover disks attached to the machine when copying configuration across old / new Clover configurations.

With the exception of: HFSPlus.efi and apfs.efi files, all the files within the driver directory are part of Clover, the two exception are taken from an actual Mac. These are the HPFS+ (from physical firmware) and APFS (from disk) drivers which are required to read the Apple formatted disk volumes.

The VBoxHfs.efi is a Clover supplied alternate to Apple HFSPlus.efi driver.

When you install Clover you can select to have all the drivers available installed  in the "/Volumes/EFI/EFI/Clover/drivers/off" directory and then selectively move the ones you need into the UEFI directory.

NOTE 1: With KVM based Macs there is no need to get into the black magic of extra kexts (kernel extensions) or ACPI / DSDT (Advance Configuration & Power Interface / Differentiated System Description Table) tweeking.

NOTE 2: The lastest version of Clover are 64-Bit UEFI only, in prior version there were both sets of 32-Bit and 64-Bit EFI files. In my Clover:

  • V2.4 Rel. 44509 - the EFI driver files are in "/Volumes/EFI/EFI/CLOVER/drivers64UEFI" while
  • V2.5 Rel. 5070 - the EFI driver files are in "/Volumes/EFI/EFI/CLOVER/drivers"

So it appears that the need to specify 64-Bit version has gone.

To covert VMWare EFI Boot disk to QEMU qcow2 one:

# qemu-img convert -f vmdk -O qcow2 WMWARE-EFI.vmdk QEMU-EFI.qcow2

QEMU Virtual Machine Setup

To run macOS machine with QEMU / KVM requires very specific set up, summarised here:

  • Q35 - Virtual Machine (v 2.11, 2.12 work) [Also I have tested with v 3.1 post installation boot up , but not with a new install]
  • OVMF - Firmware which has associated NVRAM storage blob (OVMF_VARS.fd). Each bootable macOS machines needs to have its own NVRAM storage device
  • e1000-82545em - network device, connected to bridge br10 in below example [or vmxnet3 (VMWare virtual NIC driver) testing within 10.12 Sierra onwards]
  • SATA - Disk and CDROM only, in this example there are 2 disks attached, the Clover Boot disk and the virtual MacOS one
  • USB Keyboard
  • EvTouch USB Graphics Tablet
  • VGA Graphics

In addition the Mac VM is very sensitive to CPU type and features, for Catalina here are details of CPU configuration:

<cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Penryn</model>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='ssse3'/>
    <feature policy='require' name='sse4.2'/>
    <feature policy='require' name='popcnt'/>
    <feature policy='require' name='avx'/>
    <feature policy='require' name='aes'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>

The macOS machine also needs to get its special applesmc configuration:

    <qemu:arg value='-device'/>
    <qemu:arg value='isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'/>

NOTE: See "Mac OS X Internals - A Systems Approach" Bonus chapter to get above programmically on your Mac.

My approach is to use the Virtual Machine Manager to create initial configuration, then start machine to save the configuration and then edit the configuration to set the detailed CPU, Video & OVMF_VARS.fd configuration:

  • "sudo virsh edit MAC-NAME" - to edit final configuration details.

As part of initial boot do an escape at Clover startup and change the Clover resolution value to match the config-plist value (in this example: 1024x768)

Here is example of complete configuration for (Catalina Mac):

# virsh dumpxml mac-test-macos 
<domain type='kvm' id='37' xmlns:qemu=''>
  <memory unit='KiB'>37257216</memory>
  <currentMemory unit='KiB'>37257216</currentMemory>
  <vcpu placement='static'>4</vcpu>
    <type arch='x86_64' machine='pc-q35-2.12'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <bootmenu enable='yes'/>
    <vmport state='off'/>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Penryn</model>
    <topology sockets='2' cores='2' threads='1'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='ssse3'/>
    <feature policy='require' name='sse4.2'/>
    <feature policy='require' name='popcnt'/>
    <feature policy='require' name='avx'/>
    <feature policy='require' name='aes'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/USER/mac.test.macos/clover-boot-25k-r5070.qcow2'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='sdc' bus='sata' tray='open'/>
      <boot order='1'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/USER/mac.test.macos/mac-hd1-01.qcow2'/>
      <target dev='sdk' bus='sata'/>
      <alias name='sata1-0-4'/>
      <address type='drive' controller='1' bus='0' target='0' unit='4'/>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    <controller type='sata' index='1'>
      <alias name='sata1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    <controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x11'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x12'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x13'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    <interface type='bridge'>
      <mac address='XX:XX:XX'/>
      <source bridge='br10'/>
      <target dev='vnet31'/>
      <model type='e1000-82545em'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    <serial type='pty'>
      <source path='/dev/pts/30'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      <alias name='serial0'/>
    <console type='pty' tty='/dev/pts/30'>
      <source path='/dev/pts/30'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
      <address type='usb' bus='0' port='2'/>
    <input type='mouse' bus='ps2'>
      <alias name='input2'/>
    <input type='keyboard' bus='ps2'>
      <alias name='input3'/>
    <graphics type='spice' port='5932' autoport='yes' listen=''>
      <listen type='address' address=''/>
      <image compression='off'/>
      <model type='vga' vram='65536' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='3'/>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='4'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'/>

Once you have completed editing using virsh, exit the editor and restart machine with all the correct configuration.

Once you have got a running macOS running, then for each new machine just copy the working Clover Boot disk and OVMF_VARS.fd files into new location for this new machine and then on initial bood go into Clover Configurator to edit the configuraton (serial numbers and UUID) so that it has its own unique identify.

Optionally Add PCI Passthrough

If you you have configured you kernel correctly (see above grub configuration), then this is trivial. Simple grab the PCI Resource via Virtual Machine Manager GUI and reboot the machine.

I have successfully attached: SmallTree 10Gbe and Areca RAID controllers to QEMU / KVM Virtual Mac.

To and From Physical and Virtual Macs

It is relatively easy to move physical Mac to Virtual and back again. The tools for this include:

  • qemu-img - to convert images
  • Carbon Copy Cloner - to copy bootable images
  • OS Bless Command - to re-bless a HD volume to make it bootable

References & Links

After much effort and testing based on the following information from: Kraxel's, Kholia, Gordon Turner and Clover sites:

Kraxel - did a lot of ground work to get MacOS running on KVM, most of which has flowed into upstream KVM / QEMU and so this is historical now

Kholia - keeps onging testing and configuration tweeking information as MacOS goes through its release cycles

Gordon Turner - has some useful pointers that where easier to follow than some of Kholia's instructions

Clover - provides the MacOS EFI implementation tweeks required to bt MacOS running on KVM

Clover Configurator - only availeable as a binary executable, so you have to make a trust decision on whether to use it

My earlier notes on Insanely Mac

Mac OS X Internals - A Systems Approach by Amit Singh provides most of what you need to know about EFI and OS X (now macOS) boot process and has bonus chapter on SMC keys needed to boot OS X