Effortless FreeBSD VM Control with bhyve and vm-bhyve

(2024-10-26):

The bhyve hypervisor on FreeBSD is an impressive tool that brings lightweight, efficient virtualization capabilities directly into the FreeBSD ecosystem. It’s a powerful type 2 hypervisor that allows FreeBSD users to run virtual machines with minimal overhead, making it an excellent choice for those who value performance and stability. I find bhyve especially compelling for running various Unix-like operating systems like FreeBSD itself, Linux, and even Windows, thanks to UEFI support.

In the past, I already wrote many blog posts about virtualization and hypervisor solutions where I also often mentioned bhyve but also the lacks around bhyve. However, bhyve is still an amazing way to virtualize your workloads by running VMs of any operating system. Recently I got ask how I usually manage bhyve and if I could provide a small howto.

To manage bhyve VMs more easily, I rely on vm-bhyve, which streamlines the creation, configuration, and operation of virtual machines on the cli. If you’re looking for a GUI or web ui tool, solutions like BVCP (bhyve-webadmin) or ClonOS may fit your needs. With its intuitive command set, vm-bhyve makes it easy to spin up, start, and stop VMs. It supports a straightforward configuration file format, which simplifies setting up new virtual instances.

This is particularly useful when managing a larger number of virtual environments or deploying machines across multiple nodes but also on single node setups.

The network configuration within vm-bhyve also shines, as it lets me set up bridges that support VLANs, complete with automatic device creation. This flexibility, combined with its ZFS integration, makes managing storage both easy and efficient, leveraging FreeBSD’s native file systems capabilities to enhance VM performance and reliability.

Running various guests, from Linux to the BSD family (NetBSD, OpenBSD, and of course, FreeBSD), is also straightforward. vm-bhyve automates the console device assignments, giving me easy access to each VM’s console output.

With support for VNC graphics and tmux in later versions, managing graphical guests or keeping consoles open in multiple sessions has become even more convenient. And since the tool integrates smoothly with rc.d, I can rest assured that VMs start up and shut down gracefully along with the host system.

A few additional packages are necessary to fully utilize vm-bhyve’s features. For example, ca_root_nss is included in the FreeBSD port to prevent SSL errors when downloading FreeBSD ISO files. The grub2-bhyve package is required for Linux guests or any system that relies on a GRUB bootloader, while bhyve-firmware enables UEFI guest support. Finally, tmux is a great option for console access if I prefer it over the default cu/nmdm.

HowTo

The “hot section” of setting up bhyve on a new FreeBSD host is an exciting stage, where we get to install and configure the essential tools and set up the system for virtualization. With just a few straightforward commands, we can transform a FreeBSD host into a fully operational virtualization platform, leveraging bhyve as our hypervisor and vm-bhyve for VM management.


pkg install -y wget vm-bhyve
zfs create zroot/vm
sysrc vm_enable="YES"
sysrc vm_dir="zfs:zroot/vm"
vm init
vm switch create public
vm switch add public vtnet0
wget https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/14.1/FreeBSD-14.1-RELEASE-amd64-bootonly.iso
vm create boxybsd581
vm install boxybsd581 FreeBSD-14.1-RELEASE-amd64-bootonly.iso
vm console boxybsd581

This doesn’t only look pretty easy and quick – it is! This way, you can spawn up your first VM guest on a shiny new FreeBSD system in just 120 seconds. You don’t believe? Check out my live demonstration at YouTube.

Configuration

Once we’ve set up our FreeBSD host with bhyve and vm-bhyve, making modifications to a virtual machine’s configuration is straightforward. The configuration files for each VM are conveniently located in the storage directory we initially defined (e.g., /zroot/vm), under a subdirectory matching the VM’s name.

To modify a VM’s settings, all we need to do is edit its configuration file, typically located at /zroot/vm/$vm_name/$vm_name.conf. This plain-text file defines various resource parameters such as CPU cores, memory allocation, network interfaces, and storage devices.


root@virt83:~ # cat /zroot/vm/boxybsd581/boxybsd581.conf 
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"
uuid="78d2433c-9373-11ef-8180-bc2411326645"
network0_mac="fc:fc:fc:fc:fc:fc"

Conclusion

In my experience, FreeBSD and its native hypervisor, bhyve, make running virtualization workloads incredibly efficient, reliable, and straightforward. FreeBSD’s simplicity is one of its biggest strengths, and bhyve’s tight integration with the base system allows me to spin up virtual machines with minimal fuss.

One of the standout features for me has been the ZFS filesystem. With instant snapshots, strong data integrity, and efficient storage usage, ZFS provides a level of reliability that’s hard to match. The snapshotting feature alone has saved me countless times.

Networking is another area where FreeBSD shines. Using the pf firewall, I can set up complex firewalling rules while maintaining clarity and control. The networking stack integrates smoothly with bhyve, making segmentation, switching, and NAT intuitive and reliable.

Overall, the combination of FreeBSD, bhyve, ZFS, and pf firewalling gives me a virtualization environment that’s fast, reliable, and secure. Everything is integrated, efficient, and predictable which is exactly what I want from a virtualization platform.