Armbian guys are great in what they do with the sunxi64 kernel for the Pinebook. I have been experimenting with the Pinebook for 2 months now and so far I have used the native Ubuntu 18.04 aarch64 distro, NetBSD -current from Jared McNeill (http://www.invisible.ca/arm/) and the Armbian for Pinebook (https://www.armbian.com/pinebook-a64/)
I have so far best experience with the Armbian pinebook-a64 Linux distribution and here I would like to describe the process how to get full KVM support in the Armbian Bionic based on the 4.19.x kernel
Unfortunately the current 4.19.2-sunxi64 kernel does not have KVM virtualisation support enabled so any attempt on running aarch64 virtual machines in QEMU on the Pinebook will be horribly slow.
Installation of the Armbian Stretch 4.19.x distribution is straight forward, and I have used the image that I have transferred to a microSD card 64BG (100 mb/s as minimum speed). All this can be easily done on the current Linux machine via following command
x86_64 LAPTOP $ wget https://dl.armbian.com/pinebook-a64/Debian_stretch_next.7z $ su # dd if=Armbian_5.69_Pinebook-a64_Debian_stretch_next_4.19.13.img of=/dev/sd(X) <- check what your USB microSD adaper is
Once the image is transferred, please follow the official documentation how to setup Networking and all other things, which I’m not going to cover here. It is straight forward.
So once we have the Pinebook booted and connected to the internet, make sure you get all packages updated
aarch64 PINEBOOK # apt-get update # apt-get upgrade
we can go on and install latest Qemu, so for this we need to enable the testing repository in /etc/apt/sources.list
aarch64 PINEBOOK root@pinebook:/etc/apt# cat sources.list deb http://httpredir.debian.org/debian testing main contrib non-free #deb-src http://httpredir.debian.org/debian stretch main contrib non-free deb http://httpredir.debian.org/debian stretch-updates main contrib non-free #deb-src http://httpredir.debian.org/debian stretch-updates main contrib non-free deb http://httpredir.debian.org/debian stretch-backports main contrib non-free #deb-src http://httpredir.debian.org/debian stretch-backports main contrib non-free deb http://security.debian.org/ stretch/updates testing main contrib non-free #deb-src http://security.debian.org/ stretch/updates main contrib non-free deb http://httpredir.debian.org/debian jessie main contrib non-free
So next we can install qemu-system which will pull all softmmu suport (x86,sparc,ppc,mips,arm) but for accelerated qemu emulation we obviously need only the qemu-system-arm (this includes both qemu-system-arm and qemu-system-aarch64) We will also need to install qemu-utils and qemu-efi-aarch64
aarch64 PINEBOOK # apt-get install qemu-system qemu-efi-aarch64
So now we are ready to build our customized Armbian Stretch kernel 4.19.x based on mainline, since the stock one does not include KVM support.
So for this I have followed this guide https://docs.armbian.com/Developer-Guide_Build-Preparation/
What we will need is the following:
- Host x86_64 Linux system capable of running VirtualBox (6.0) with 8GB RAM and at least 50 GB disk space available
- Configured and working VirtualBox (I have used 6.0 https://download.virtualbox.org/virtualbox/6.0.0/virtualbox-6.0_6.0.0-127566~Ubuntu~bionic_amd64.deb )
- Exactly this version of Ubuntu 18.04 http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/mini.iso
I have used the following config on my Linux Mint 19.1 – make sure you create a large enough VDI disk to hold all the build files, minimum is 20 GB, 50 GB is recommended.
Also Im using the following host-only-adapter network setup so that I can access the VM from my host and vice-versa easily via ssh/scp
For this to work I have a following network configuration script I run after I start VirtualBox manager (to get the vboxnet0 interface registered first)
x86_64 LAPTOP root@panasonic:/home/user# cat networking.sh #Setup tap and bridge tunctl -t tap0 -u user ifconfig tap0 up brctl addbr br0 brctl addif br0 vboxnet0 brctl setfd br0 0 ifconfig br0 10.0.2.2 netmask 255.255.255.0 broadcast 10.0.2.255 up brctl addif br0 tap0 vboxnet0 ifconfig tap0 0.0.0.0 sysctl net.ipv4.ip_forward=1 iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE
Internal IP address for the Guest VMs are always static in the following format
IP: 10.0.2.10-200 Gateway: 10.0.2.2 DNS: 8.8.8.8 or 1.1.1.1
Setup the OpenSSH server and configure the network on the Guest Ubuntu VM accordingly.
So once we have everything ready we can start preparing the Guest Ubuntu VM for the kernel build
Login, elevate to root and run
VirtualBox Ubuntu GUEST
# apt-get -y -qq install git
# git clone --depth 1 https://github.com/armbian/build
# cd build
And run the compile.sh script
VirtualBox Ubuntu GUEST # ./compile.sh
Select to build U-boot and Kernel packages
Choose to show Kernel options so that we can enable the KVM support
Choose Pinebook-a64
After a while you should come to the Kernel menuconfig
Make sure to set the following Virtualisation option on (space bar)
And the following subsection should be enabled as built-on and modules
After some time the kenrel packages would be made available in the following directory which we will scp out of the VirtualBox VM and transfer to the Pinebook via scp
Back on the Pinebook we need to install these packages via apt
aarch64 PINEBOOK -rw-r--r-- 1 user user 44152 Jan 11 23:36 linux-dtb-next-sunxi64_5.70_arm64.deb -rw-r--r-- 1 user user 10422904 Jan 11 23:35 linux-headers-next-sunxi64_5.70_arm64.deb -rw-r--r-- 1 user user 16571572 Jan 11 23:35 linux-image-next-sunxi64_5.70_arm64.deb -rw-r--r-- 1 user user 320580692 Jan 11 23:36 linux-source-next-sunxi64_5.70_all.deb -rw-r--r-- 1 user user 246032 Jan 11 23:35 linux-u-boot-next-pinebook-a64_5.70_arm64.deb # dpkg -i linux-dtb-next-sunxi64_5.70_arm64.deb # dpkg -i linux-headers-next-sunxi64_5.70_arm64.deb # dpkg -i linux-image-next-sunxi64_5.70_arm64.deb
Reboot the Pinebook to load the new KVM enabled Kernel, once up we should verify we are running the new Kernel
aarch64 PINEBOOK root@pinebook:/home/user# uname -a Linux pinebook 4.19.13-sunxi64 #5.70 SMP Sat Jan 12 00:10:12 CET 2019 aarch64 GNU/Linux root@pinebook:/home/user#
Next we prepare the virtualisation environment where we want to run Centos 7.6
Before we actually load QEMU on the Pinebook we need to modify slightly the stock /etc/qemu-ifup script as follows
#! /bin/sh
# Script to bring a network (tap) device for qemu up.
# The idea is to add the tap device to the same bridge
# as we have default routing to.
# in order to be able to find brctl
PATH=$PATH:/sbin:/usr/sbin
ip=$(which ip)
if [ -n "$ip" ]; then
ip link set "$1" up
else
brctl=$(which brctl)
if [ ! "$ip" -o ! "$brctl" ]; then
echo "W: $0: not doing any bridge processing: neither ip nor brctl utility not found" >&2
exit 0
fi
ifconfig "$1" 0.0.0.0 up
fi
switch=$(ip route ls |
awk '/^default / {
for(i=0;i<NF;i++) { if ($i == "dev") { print $(i+1); next; } }
}'
)
switch=br0
# only add the interface to default-route bridge if we
# have such interface (with default route) and if that
# interface is actually a bridge.
# It is possible to have several default routes too
for br in $switch; do
if [ -d /sys/class/net/$br/bridge/. ]; then
if [ -n "$ip" ]; then
ip link set "$1" master "$br"
else
brctl addif $br "$1"
fi
exit # exit with status of the previous command
fi
done
echo "W: $0: no bridge for guest interface found" >&2
And prepare the following “network” helper script to enable internet connection sharing bethween Host and Quest on the Pinebook
You will however need to install some tools first for this to work
aarch64 PINEBOOK # apt-get install uml-utilities # apt-get install bridge-utils
And then finally save below shell script, make it executable and run it
#Setup tap and bridge tunctl -t tap0 -u user ifconfig tap0 up brctl addbr br0 brctl setfd br0 0 ifconfig br0 10.0.2.2 netmask 255.255.255.0 broadcast 10.0.2.255 up brctl addif br0 tap0 ifconfig tap0 0.0.0.0 sysctl net.ipv4.ip_forward=1 iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE
Load the above script each time you boot the Pinebook if you wish to run Qemu.
On the Pinebook prepare a directory to hold some files, we will be using the linaro UEFI image
aarch64 PINEBOOK $ mkdir -p ~/KVM/centos $ cd ~/KVM/centos $ wget http://mirror.vpsnet.com/centos-altarch/7.6.1810/isos/aarch64/CentOS-7-aarch64-Minimal-1810.iso $ mv CentOS-7-aarch64-Minimal-1810.iso centos.iso $ wget http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.img.gz $ gunzip QEMU_EFI.img.gz $ qemu-img create -f qcow2 centos.img 50G $ qemu-img create -f qcow2 varstore.img 64M
And next we prepare the qemu loader shell script for this simulation in the KVM/centos working directory and make it executable
/usr/bin/qemu-system-aarch64 \ -cpu host -M virt,accel=kvm -m 1024 -nographic \ -drive if=pflash,format=raw,file=QEMU_EFI.img \ -drive if=pflash,file=varstore.img \ -drive if=virtio,file=disk.img \ -drive if=virtio,format=raw,file=centos.iso \ -net nic -net tap
This is it, and you should be able to fully emulate with almost native speed CentOS 7.6 aarch64 on the Debian 9.6 aarch64 Pinebook
Configuration of CentOS is pretty standard.
FreeBSD 13 aarch64
The above method also works for FreeBSD 13 aarch64 ( http://ftp.freebsd.org/pub/FreeBSD/snapshots/arm64/aarch64/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm64-aarch64-PINEBOOK-20190110-r342911.img.xz ) except the qemu loader is slightly different
/usr/bin/qemu-system-aarch64 \ -cpu host -M virt,accel=kvm -m 1024 -nographic \ -drive if=pflash,format=raw,file=QEMU_EFI.img \ -drive if=pflash,file=varstore.img \ -drive if=virtio,file=freebsd.img \ -net nic -net tap
Where the freebsd.img is the downloaded img file from the above URL
P.S to save up some space on the default image rm -rf /usr/lib/debug
To use secondary disk vtblk1 use this loader script (and create disk2.img qcow2 image first)
/usr/bin/qemu-system-aarch64 \ -cpu host -M virt,accel=kvm -m 1024 -nographic \ -drive if=pflash,format=raw,file=QEMU_EFI.img \ -drive if=pflash,file=varstore.img \ -drive if=virtio,bus=0,unit=0,format=raw,file=freebsd.img \ -drive if=virtio,bus=0,unit=1,format=raw,file=disk2.img \ -net nic -net tap
Here is a dmesg from FreeBSD 13 aarch64 running under KVM qemu-system-aarch64 on Debian 9.6 aarch64 Pinebook
root@pinebook:/home/user/KVM/FreeBSD# ssh 10.0.2.11 Password for root@freebsd: Last login: Sun Jan 13 22:56:36 2019 FreeBSD 13.0-CURRENT r342911 GENERIC Welcome to FreeBSD! root@freebsd:~ # dmesg ---<<BOOT>>--- KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT r342911 GENERIC arm64 FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. KLD file umodem.ko is missing dependencies FreeBSD/SMP: Multiprocessor System Detected: 1 CPUs random: unblocking device. random: entropy device external interface MAP 785d0000 mode 2 pages 64 MAP 78610000 mode 2 pages 160 MAP 786b0000 mode 2 pages 160 MAP 7bc20000 mode 2 pages 400 MAP 7bdb0000 mode 2 pages 592 MAP 4000000 mode 0 pages 16384 MAP 9010000 mode 0 pages 1 kbd0 at kbdmux0 efirtc0: <EFI Realtime Clock> efirtc0: registered as a time-of-day clock, resolution 1.000000s acpi0: <BOCHS BXPCFACP> acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) psci0: <ARM Power State Co-ordination Interface Driver> on acpi0 gic0: <ARM Generic Interrupt Controller> iomem 0x8000000-0x8000fff,0x8010000-0x8010fff on acpi0 gic0: pn 0x2, arch 0x2, rev 0x1, implementer 0x43b irqs 288 gic0: frame: 0 8020000 1 64 80 gicv2m0: <ARM Generic Interrupt Controller MSI/MSIX> mem 0x8020000-0x8020fff on gic0 generic_timer0: <ARM Generic Timer> irq 34,35,36 on acpi0 Timecounter "ARM MPCore Timecounter" frequency 24000000 Hz quality 1000 Event timer "ARM MPCore Eventtimer" frequency 24000000 Hz quality 1000 cpu0: <ACPI CPU> on acpi0 uart0: <PrimeCell UART (PL011)> iomem 0x9000000-0x9000fff irq 0 on acpi0 uart0: console (9600,n,8,1) virtio_mmio0: <VirtIO MMIO adapter> iomem 0xa000000-0xa0001ff irq 1 on acpi0 virtio_mmio1: <VirtIO MMIO adapter> iomem 0xa000200-0xa0003ff irq 2 on acpi0 virtio_mmio2: <VirtIO MMIO adapter> iomem 0xa000400-0xa0005ff irq 3 on acpi0 virtio_mmio3: <VirtIO MMIO adapter> iomem 0xa000600-0xa0007ff irq 4 on acpi0 virtio_mmio4: <VirtIO MMIO adapter> iomem 0xa000800-0xa0009ff irq 5 on acpi0 virtio_mmio5: <VirtIO MMIO adapter> iomem 0xa000a00-0xa000bff irq 6 on acpi0 virtio_mmio6: <VirtIO MMIO adapter> iomem 0xa000c00-0xa000dff irq 7 on acpi0 virtio_mmio7: <VirtIO MMIO adapter> iomem 0xa000e00-0xa000fff irq 8 on acpi0 virtio_mmio8: <VirtIO MMIO adapter> iomem 0xa001000-0xa0011ff irq 9 on acpi0 virtio_mmio9: <VirtIO MMIO adapter> iomem 0xa001200-0xa0013ff irq 10 on acpi0 virtio_mmio10: <VirtIO MMIO adapter> iomem 0xa001400-0xa0015ff irq 11 on acpi0 virtio_mmio11: <VirtIO MMIO adapter> iomem 0xa001600-0xa0017ff irq 12 on acpi0 virtio_mmio12: <VirtIO MMIO adapter> iomem 0xa001800-0xa0019ff irq 13 on acpi0 virtio_mmio13: <VirtIO MMIO adapter> iomem 0xa001a00-0xa001bff irq 14 on acpi0 virtio_mmio14: <VirtIO MMIO adapter> iomem 0xa001c00-0xa001dff irq 15 on acpi0 virtio_mmio15: <VirtIO MMIO adapter> iomem 0xa001e00-0xa001fff irq 16 on acpi0 virtio_mmio16: <VirtIO MMIO adapter> iomem 0xa002000-0xa0021ff irq 17 on acpi0 virtio_mmio17: <VirtIO MMIO adapter> iomem 0xa002200-0xa0023ff irq 18 on acpi0 virtio_mmio18: <VirtIO MMIO adapter> iomem 0xa002400-0xa0025ff irq 19 on acpi0 virtio_mmio19: <VirtIO MMIO adapter> iomem 0xa002600-0xa0027ff irq 20 on acpi0 virtio_mmio20: <VirtIO MMIO adapter> iomem 0xa002800-0xa0029ff irq 21 on acpi0 virtio_mmio21: <VirtIO MMIO adapter> iomem 0xa002a00-0xa002bff irq 22 on acpi0 virtio_mmio22: <VirtIO MMIO adapter> iomem 0xa002c00-0xa002dff irq 23 on acpi0 virtio_mmio23: <VirtIO MMIO adapter> iomem 0xa002e00-0xa002fff irq 24 on acpi0 virtio_mmio24: <VirtIO MMIO adapter> iomem 0xa003000-0xa0031ff irq 25 on acpi0 virtio_mmio25: <VirtIO MMIO adapter> iomem 0xa003200-0xa0033ff irq 26 on acpi0 virtio_mmio26: <VirtIO MMIO adapter> iomem 0xa003400-0xa0035ff irq 27 on acpi0 virtio_mmio27: <VirtIO MMIO adapter> iomem 0xa003600-0xa0037ff irq 28 on acpi0 virtio_mmio28: <VirtIO MMIO adapter> iomem 0xa003800-0xa0039ff irq 29 on acpi0 virtio_mmio29: <VirtIO MMIO adapter> iomem 0xa003a00-0xa003bff irq 30 on acpi0 virtio_mmio30: <VirtIO MMIO adapter> iomem 0xa003c00-0xa003dff irq 31 on acpi0 virtio_mmio31: <VirtIO MMIO adapter> iomem 0xa003e00-0xa003fff irq 32 on acpi0 pcib0: <Generic PCI host controller> on acpi0 pci0: <PCI bus> on pcib0 pcib0: pci_host_generic_core_alloc_resource FAIL: type=4, rid=16, start=0000000000000080, end=000000000000009f, count=0000000000000020, flags=1400 pcib0: pci_host_generic_core_alloc_resource FAIL: type=4, rid=16, start=0000000000000040, end=000000000000007f, count=0000000000000040, flags=1800 virtio_pci0: <VirtIO PCI Network adapter> mem 0x10042000-0x10042fff,0x8000000000-0x8000003fff at device 1.0 on pci0 vtnet0: <VirtIO Networking Adapter> on virtio_pci0 vtnet0: Ethernet address: 52:54:00:12:34:56 virtio_pci1: <VirtIO PCI Block adapter> mem 0x10041000-0x10041fff,0x8000004000-0x8000007fff at device 2.0 on pci0 vtblk0: <VirtIO Block Adapter> on virtio_pci1 vtblk0: 2560MB (5242880 512 byte sectors) virtio_pci2: <VirtIO PCI Block adapter> mem 0x10040000-0x10040fff,0x8000008000-0x800000bfff at device 3.0 on pci0 vtblk1: <VirtIO Block Adapter> on virtio_pci2 vtblk1: 0MB (385 512 byte sectors) acpi_button0: <Power Button> on acpi0 acpi0: Could not update all GPEs: AE_NOT_CONFIGURED cryptosoft0: <software crypto> Timecounters tick every 1.000 msec usb_needs_explore_all: no devclass CPU 0: ARM Cortex-A53 r0p4 affinity: 0 Instruction Set Attributes 0 = <AES+PMULL,SHA1,SHA2,CRC32> Instruction Set Attributes 1 = <> Processor Features 0 = <AdvSIMD,Float,EL3 32,EL2 32,EL1 32,EL0 32> Processor Features 1 = <0> Memory Model Features 0 = <4k Granule,64k Granule,MixedEndian,S/NS Mem,16bit ASID,1TB PA> Memory Model Features 1 = <> Memory Model Features 2 = <32b CCIDX,48b VA> Debug Features 0 = <2 CTX Breakpoints,4 Watchpoints,6 Breakpoints,PMUv3,Debug v8> Debug Features 1 = <0> Auxiliary Features 0 = <0> Auxiliary Features 1 = <0> Trying to mount root from ufs:/dev/ufs/rootfs [rw]... WARNING: WITNESS option enabled, expect reduced performance. lo0: link state changed to UP vtnet0: link state changed to UP