Installing KVM in Lubuntu14.04 with Nested Virtualization Support

Virtualization can be quite useful. Rather than disrupting our base system, we can create virtual machines, try things out, snapshot it, restore back and throw away once done. We can try out a new update’s stability in a virtual machine, before applying to your physical machine. Also we can run other OS’s along side with your OS of choice, like running Windows 8.1 along side of your Ubuntu installation without a reboot. In virtualization terms, our physical computer will be called as Host and the virtual machines created are called ‘Guests’. The software that provide virtualization capability to your physical computer, is called Hypervisor.

Hypervisors are categorized in to Type1 (Bare Metal) and Type2.

Type1 will be directly installed to the ‘Host’ machine and it does not requires any existing operating system to work. It will directly talk to the hardware and will manage virtual machines.  So Type1 hypervisors are special OS’s, that are specifically designed for virtualization tasks. So they provide ‘near native’ performance for virtual machines. Examples are VMWare ESXi, XenClient and XenServer.

Type2 requires an existing ‘OS’ installation and will be installed on top of it. For managing physical hardware resources, Type2 hypervisors will talk to the installed OS (Called Host OS), and Host OS will in turn talk to the hardware. So due to this double indirection, Type2 hypervisors are bit slower than Type1. Examples are VirtualBox, VMWare Workstation and VirtualPC.

There are certain OS’s that are neither Type1, nor Type2, like KVM. In fact KVM is sometime called as Type1, but it does requires an existing OS installation to work. The KVM Hypervisor module itself is implemented as a ‘Kernel Module’ and hence, it might be providing near native performance. Still user space tools are required to manage virtual machines like ‘libvirt’ and ‘qemu-kvm’, that are installed on top of the Host OS.

Whether it’s Type1 or Type2, most of the Hypervisors are designed to use ‘Hardware Assisted Virtualization’ to boost ‘Guest machine’s’ performance. In other terms they make use of your physical computer's processor virtualization extensions (AMD-V for AMD processors, and Intel-VT for Intel processors), to improve ‘Virtual Machine’ performance. This requires ‘Virtualization Extension’ to be supported by your ‘Processor’ (And it does for all recent Intel/AMD processors) and need to be enabled under your ‘BIOS’. Certain hypervisors (like Virtualbox) can work without ‘Hardware Assisted Virtualization’, and that scenario is called ‘Full Virtualization’ where performance will be much lower.

Some of the Hypervisors strictly requires ‘Hardware Assisted Virtualization’ to be available and enabled in BIOS and is recommended as well. Example XenClient, VMWare ESXi.

OK, that’s all about virtualization and hypervisors. Now consider an extreme scenario. You want to virtualize a Hypervisor itself! That means you would like to install a Hypervisor inside an Hypervisor and the nested Hypervisor demands ‘Virtualization Extensions’. This scenario requires the Hypervisor running on the Host, should be able to pass through the ‘Virtualization Extension’ (AMD-V, Intel-VT) to the nested Hypervisor (i.e The Guest Hypervisor, which is running as a virtual machine). This is called ‘Nested Virtualization’. Using this technique, you can test various Hypervisors in virtual environment, before the actual deployment to checkout the features and limitations. You can try create virtual machines inside the nested hypervisor, which itself is a virtual machine. Sounds interesting right?

For common purposes, we heavily use Virtualbox (A Type2 Hypervisor), as it is quite simple to setup and have a very friendly GUI for managing virtual machines. But it lacks on ‘Nested Virtualization’ support for the guests.

Recently we’d to virtualize a Bare Metal hypervisor, named ‘XenClient’ that mandates ‘Processor Virtualization Extensions’ to be available on it’s virtual host (i.e Virtual machine, on which ‘XenClient’ got installed).

XenClient is a ‘Bare Metal’ Type-1, hypervisor specifically designed for Desktops and Laptops. Though the hardware support is somewhat limited for older version, the compatibility base is now getting better with the latest versions. XenClient requires ‘Processor Virtualization Extensions’ to be available and it has one more additional requirement, It does requires hard disk to be attached to SATA Controller. It does not support IDE controller for hard disks.

So to virtualize XenClient, we cannot use Virtualbox, as it does not support nested virtualization. We cannot use ‘VMWare Workstation’ either, as it does not support SATA controllers, though it supports nested virtualization.

As per our experience, we found that ‘KVM’ (Kernel Virtual Machine) is the best option for such nested virtualization scenarios, where you want to run a Hypervisor as a guest inside a Hypervisor. KVM does allows nested virtualization through some minor tweaks. It also supports SATA controllers for hard disks. KVM is by default available in mainline Linux kernel. The below steps explains, setting up KVM in Lubuntu14.04 host with Nested Virtualization features.

1. Check whether, your processor support ‘Virtualization Extensions’? (Intel-VT/AMD-V)

Run the following command in the terminal.

“egrep -c '(vmx|svm)' /proc/cpuinfo”

If 0 it means that your CPU doesn't support hardware virtualization. If 1 or more it does - but you still need to make sure that virtualization is enabled in the BIOS.

Alternatively, you may execute: “kvm-ok”. This should return the below.

INFO: /dev/kvm exists
KVM acceleration can be used

2. Install KVM and Management Tools

Use the below command line to install the Qemu emulation layer and the virtual machine manager.

“sudo apt-get install qemu-kvm libvirt-bin virt-manager bridge-utils”

Add the current user to the ‘libvirtd’ group.

“sudo adduser `id -un` libvirtd”

3. Enable Nested Virtualization In KVM

By default nested virtualization is not enabled in KVM, It needs to be activated explicitly. First try check its already enabled by you previously, by issuing the below command;

cat /sys/module/kvm_intel/parameters/nested

If the output is ‘N’. Then enable it by issuing the below command.

echo 'options kvm_intel nested=1' >> /etc/modprobe.d/qemu-system-x86.conf

We’re done setting up KVM. Now only thing remember is to enable ‘Nested Virtualization’ while creating a virtual machine using ‘Virt-Manager’, that is explained in the next step.

3. Created a virtual machine with Nested Virtualization Support.

In this exercise, we’re installing ‘XenClient’ (A Type1, Bare Metal Desktop Hypervisor) as a guest in KVM. Create a virtual machine and install ‘XenClient’. Before install, ensure the below settings, for the virtual machine.

Ensure ‘vmx’ settings as ‘require’. This enables the nested virtualization available to the KVM guest.

Below are the settings specifically required for ‘XenClient’.

Ensure, Virtual disk is attached to ‘SATA’ controller.

Ensure, Video model as ‘VMVGA’. NB: This is important, ‘XenClient’ only seems to detect this video model only under KVM.

Here you are! Below figure shows our successfully installed and running ‘XenClient’ as a KVM Guest.

Note: If you’re getting the below error while starting a nested guest, inside XenClient, check XenClient system requirements and verify your hardware compatibility. Your processor, chipset, and BIOS settings should support VT-x and VT-d (For inter processors) and must be enabled.

4. Appendix.

Read more on KVM installation in Ubuntu here.

Read more on configuring Nested KVM under Ubuntu14.04 here.