When I think about bhyve Live Migration, it’s something I encounter almost daily in my consulting calls. VMware’s struggles with Broadcom’s licensing issues have been a frequent topic, even as we approach the end of 2024. It’s surprising that many customers still feel uncertain about how to navigate this mess. While VMware has been a mainstay in enterprise environments for years, these ongoing issues make customers nervous. And they should be – it’s hard to rely on something when even the licensing situation feels volatile.

Now, as much as I’m a die-hard FreeBSD fan, I have to admit that FreeBSD still falls short when it comes to virtualization – at least from an enterprise perspective. In these environments, it’s not just about running a VM; it’s about having the flexibility and capabilities to manage workloads without interruption. Years ago, open-source solutions like KVM (e.g., Proxmox) and Xen (e.g., XCP-ng) introduced features like live migration, where you can move VMs between hosts with zero downtime. Even more recently, solutions like SUSE Harvester (utilizing KubeVirt for running VMs) have shown that this is now an essential part of any virtualization ecosystem.

Whenever I talk to customers about possible alternatives to VMware, I find myself saying, “There’s nothing that can replace VMware one-to-one!” And it’s unfortunately the truth. That’s why it’s so important to understand their requirements fully before suggesting any solutions. Nine times out of ten, live migration is a critical or even very critical requirement. And as soon as we get to that point, it becomes clear that FreeBSD just isn’t a realistic option for most enterprises right now – at least not for virtualization environments with enterprise requirements.

Sure, I’ve been able to migrate a few systems to FreeBSD based solutions, but these have always been single-node setups. That’s fine for some use cases, but in today’s enterprise world, the idea of telling a customer they need to shut down their VMs just to perform maintenance on the host is absurd. Downtime, even for short periods, is unacceptable. Even if you’re just copying a VM’s disk to another host, shutting it down, and syncing deltas before bringing it back online, the downtime is still too long by 2024 standards. All established connections would drop. Imagine dropping the VPN connections, (huge) file transfers and all other important sessions. That’s simply not a viable solution.

We were finally starting to make progress with bhyve and live migration in the FreeBSD world, and it’s a huge relief. For a long time, I’ve felt that we needed to shift away from the mindset of “it fits for me.” Sure, FreeBSD works great for those of us who are die-hard fans, but the truth is, there are millions of people out there with different requirements, especially in enterprise environments. We can’t just focus on what works for us personally – if we want FreeBSD to grow and become a serious competitor to Linux-based virtualization solutions, we need to think bigger. I’ve already written a blog post about this, sharing my thoughts on how we can make FreeBSD more attractive to new users and meet the demands of modern IT infrastructure.

Luckily, there are some really talented people working to address these gaps. Elena Mihăilescu, Mihai Carabaș, Oleg Ginzburg, and Oleg Minin (Oleg G., and Oleg. M. in cbsd (cbsd live migration)/ ClonOS) are tackling the tough parts of making live migration a reality for FreeBSD. Elena and Mihai presented the latest updates at BSDCan 2019 in Ottawa, and their work gives us a clearer picture of how things are evolving. Their paper provides fresh insights into the concepts behind moving a guest from one host to another, covering cold, warm, and live migrations.

Live migration, of course, is the most critical for enterprises. It allows you to move a running VM from one host to another without any downtime, and it works by migrating the memory in rounds while the guest is still running. In the final round, the source VM is stopped, any remaining memory is copied over, along with the CPU and device state, and then the VM is restarted on the destination host. It’s a delicate process, and memory migration is always the most complex part, requiring multiple rounds to ensure a smooth transition.

(You can find the full paper here)

One of the techniques they use is a page fault-based approach. And here’s how it works: first, the source VM is stopped and its CPU and device state are copied to the destination host. Then, the VM is started on the new host, but instead of copying all the memory at once, the system waits for a page fault to occur. When a page fault happens (meaning the VM tries to access a part of memory that hasn’t been transferred yet), that memory page is copied over at that moment. This method helps reduce the downtime and keeps the VM running as smoothly as possible during the migration.

Like with any live migration technology, the memory migration itself is the trickiest part. Doing it efficiently and without interrupting the running system requires careful orchestration, which is why it happens in multiple rounds. But with this ongoing work, we’re starting to see real progress, and it’s a huge step toward making FreeBSD a serious contender in the virtualization space.

Usage

How can this now be used? This is achieved by using the newly introduced parameter –migrate-live and can also easily be used.

# Run a VM
bhyve <options> vm_src

# Receive VM to be migrated
bhyve <options> -R src_IP,port vm_dst

# Send/Start VM to migrate
bhyvectl–migrate-live=dst_IP,port vm_src

Limitations

As exciting as the new developments in bhyve and live migration are, we have to be realistic about the current limitations. From my own perspective, while we’ve made progress, there are still several challenges that need to be addressed before FreeBSD and bhyve can truly compete in the enterprise virtualization space and also the slow time of progress. Don’t get me wrong here, it’s a huge process for such a small of group and the work is really appreciated but the overall progress is not comparable with other solutions and it got quiet – very quiet around this. Also, there’re still several constraints…

One of the biggest constraints is that live migration currently only works for the low memory segment – specifically for VMs with less than 3GB of RAM. In modern enterprise environments, where VMs often run workloads demanding far more memory, this is a significant limitation. A vast number of enterprise use cases rely on large memory footprints, and not being able to migrate VMs with more than ~ 14 GB of RAM (see the papers from Asia BSDCon 2023) is a major roadblock to adoption. Luckily, this is already a huge improvement to the previously mentioned 3 GB at the BSDCan 2019. Enterprises need the flexibility to migrate resource-intensive VMs, and without that, FreeBSD is missing a crucial piece of the puzzle. And while this sounds harsh, this simply reflects the current situation in FreeBSD, bhyve and market situation and does not mean that anyone is doing anything wrong – no, it’s completely the opposite! We must be aware that these guys are doing an absolutely amazing job with very complex things behind the scenes!

Another limitation is the ongoing issues with memory corruption, which seem to be tied to bhyve’s handling of threads for network and disk operations during live migration. Memory corruption is a serious problem, especially in production environments where reliability and stability are paramount. This kind of issue can cause unpredictable VM behavior or even crashes, which makes it difficult to confidently use bhyve for mission-critical workloads. Until these stability issues are ironed out, it will be hard to convince enterprises to fully trust FreeBSD as a virtualization platform.

Additionally, live migration was only possible with wired memory. This means that only pages locked into physical memory can be migrated, because if memory is swappable, there’s a risk that pages could be swapped out to disk mid-migration, leading to inconsistencies and failures during the process. While wired memory ensures that the pages are always available for migration, it’s not an ideal solution because it limits flexibility. In large environments where memory management needs to be highly dynamic, the reliance on wired memory becomes a bottleneck. Enterprises expect virtualization platforms to manage memory efficiently, including handling memory swaps and freeing up resources as needed – something that FreeBSD’s current live migration setup can’t fully support yet. However, also here a huge step forwards has been done according to the papers from the Asia BSCCon 2023.

I’m optimistic because these advancements are exactly what FreeBSD needs to become more appealing to a broader audience. We’re no longer just talking about theoretical ideas – we’re seeing real, actionable steps that could change the game. With this feature, users will be able to perform live migrations of VMs without needing to shut them down anymore. It’s a critical step in closing the gap between FreeBSD’s bhyve and the other major players in the virtualization market, such as KVM or VMware.

Imagine what this could mean for FreeBSD and bhyve if they were widely adopted in the enterprise virtualization market. More and more people would take notice of what FreeBSD has to offer, not just in terms of virtualization, but as an entire ecosystem. As interest grows, so too would the number of people using FreeBSD for a wide variety of use cases; far beyond what it is traditionally known for.

This wider adoption would create a snowball effect. Once people start using FreeBSD in their virtualization environments, they’ll naturally begin exploring its potential for other areas. They would experiment with FreeBSD in containers, networking, storage, and beyond. This growing curiosity would lead to an influx of developers and contributors eager to build solutions that address new challenges—especially in areas where there are currently no viable alternatives.

A great example of how this could unfold is what we’ve seen with Podman running on FreeBSD. The introduction of containers through Podman attracted a wave of interest from developers who were traditionally more focused on Linux environments. The same could happen with bhyve as live migration becomes more robust. Once people realize the power and flexibility of FreeBSD for virtualization, they’ll start developing additional features and solutions that extend its reach into new markets. While you can find many great HowTo’s for the general usage like from vermaden, you can also already find several great projects going into the right direction like ClonOS or BVCP (bhyve-webadmin) which could be the underlying base of an upcoming solution and fill the current gaps.

This broader engagement would be crucial for FreeBSD’s growth. It would lead to the development of enterprise-grade solutions that fulfill current gaps—things like advanced VM orchestration, better storage solutions, and integrated security features. These solutions, in turn, would make FreeBSD more attractive to enterprises that are currently dependent on Linux-based or proprietary solutions.