Some journeys don't end where you expect them to. Sometimes they bring you right back to where everything began, only with a lot more experience and a much better ecosystem waiting for you.
That's exactly what is happening with BoxyBSD.
When I first launched the project, the entire infrastructure was powered by FreeBSD and the bhyve hypervisor. It felt like the natural choice for a platform dedicated to promoting BSD operating systems. The stack was lightweight, stable, and aligned perfectly with the philosophy behind BoxyBSD. Unfortunately, the surrounding ecosystem simply wasn't ready for the level of automation and operational maturity I wanted to achieve.
As BoxyBSD continued to grow, I needed features such as seamless live migrations, centralized cluster management, and fully automated provisioning. At that time, Proxmox offered a much more complete solution. Although the migration away from FreeBSD was not an easy decision, it allowed me to transform BoxyBSD into a highly automated platform capable of operating across multiple locations with minimal manual intervention.
Ironically, moving to Proxmox VE meant building many of the features I had expected to already exist. Over the years I developed a significant amount of tooling around Proxmox to bridge those gaps. ProxLB brought VMware DRS like dynamic resource scheduling to Proxmox years before the platform introduced its own scheduler. ProxCLMC implemented an Enhanced vMotion Compatibility equivalent, allowing migrations across heterogeneous CPU generations, just like ProxSnap added a centralized way of snapshot management to Proxmox clusters. I also developed multiple Ansible modules to automate cluster deployment, creation & joining as well as node management for certificate and subscription management, just next to automate storage integrations of various storage types. I also created automated patching workflows (ProxPatch), and even maintained source code patches for Proxmox itself. One particularly memorable example was adding support for tagged VLANs during unattended installations, a feature that was essential for my deployment model but unavailable at the time.
Those projects eventually became open source and are now used by many people far beyond the BoxyBSD infrastructure. Looking back, the temporary move to Proxmox unexpectedly resulted in a whole ecosystem of tools that contributed back to the community. ProxLB quickly became a very interesting and largly used tool which caused questions in enterprises, if this could be managed by a single person for a longer time. After my ProxLB project even got showcased on the dutch CT magazine and the Dutch Proxday by the Tuxis, I gifted this project to the credativ GmbH for a better long-term maintainibility by a team. In the meantime, I got somehow part of the PegaProx project which adds all the features to a centrlaized web-ui which became similar to the vCenter - just for the Proxmox ecosystem. If all of this sounds interesting, you can find more about all those things in my blog post HOW BOXYBSD BOOSTED THE PROXMOX ECOSYSTEM WITH FREE VPS HOSTING.
While all of this was happening, I never stopped following the progress of another project that had quietly been evolving in the BSD world: Sylve.
Hayzam's work on Sylve immediately caught my attention because it pursued a very similar vision to Proxmox, but built around FreeBSD and bhyve. Instead of treating bhyve as just another hypervisor, Sylve provides a modern management platform with clustering, automation, and a user experience that feels immediately familiar to Proxmox administrators while remaining deeply rooted in the BSD ecosystem. During the first development times, I already covered Sylve in one of my blog posts.
With every release, Sylve matured rapidly. Once it finally reached a stable release and became available through the official FreeBSD pkg and ports tree, it became clear that the ecosystem had reached the point I had been waiting for. After several years running BoxyBSD on Proxmox, it was finally time to come home.
This article documents that journey: migrating the complete BoxyBSD infrastructure back to its BSD roots, replacing Proxmox with FreeBSD, bhyve, and Sylve, and exploring what the modern BSD virtualization ecosystem is capable of today.
Challenges
Moving BoxyBSD back to bhyve was much more than simply replacing one hypervisor with another. In fact, the hypervisor itself was probably the easiest part of the whole migration. The real challenge was everything surrounding it. During the years on Proxmox VE, the entire infrastructure had evolved around Linux. Automation, monitoring, deployment, provisioning, and operational workflows had all been built with Proxmox as the central component. Returning to FreeBSD meant revisiting almost every single layer of the infrastructure stack.
Fortunately, this wasn't the first time I had built such an environment. The original versions of BoxyBSD were already running on FreeBSD with bhyve before I switched to Proxmox. While this meant many design decisions were already clear, the ecosystem had changed significantly over the years. New tools had become available, others had disappeared, and operational requirements had grown considerably. BoxyBSD today hosts hundreds of virtual machines across multiple locations, making the migration far more demanding than the original deployment ever was.
The migration therefore became a complete redesign rather than a simple replacement. Every component had to be evaluated individually. Some software could simply be reused, others required adjustments, while entire parts of the automation ecosystem had to be rewritten from scratch. At the same time, the migration had one absolute requirement: existing users should not notice anything. There should be no lengthy maintenance windows, no service interruptions, and certainly no requests asking users to recreate their virtual machines.
Hardware Resources
Infrastructure migrations are usually limited by one thing: hardware. If there are no spare resources available, every migration quickly becomes a game of moving workloads around while trying to free enough capacity for the next step. Fortunately, the timing of this migration could hardly have been better.
Over the previous months, BoxyBSD had received additional sponsored hardware from several partners. At the same time, existing nodes had enough remaining capacity to temporarily consolidate workloads. This combination provided exactly the flexibility that was required for such a large-scale migration.
Where additional hardware was available, the approach was straightforward. New servers were installed with FreeBSD and Sylve from scratch, integrated into the monitoring infrastructure, configured through Ansible, and prepared for production use. Since these systems started empty, they could be tested extensively before accepting their first production workloads.
Not every Point of Presence (POP) was fortunate enough to have spare hardware available, though. In these locations, another strategy was necessary. Existing Proxmox VE clusters were consolidated by moving virtual machines between hosts using Proxmox's live migration capabilities. Since ProxLB was already responsible for automated workload placement and evacuation, these migrations could largely be orchestrated automatically. Users never noticed that their virtual machines had been moved to different physical hosts.
Once enough virtual machines had been consolidated onto fewer nodes, entire servers became available. Those systems were then completely reinstalled with FreeBSD, transformed into Sylve hypervisors, and added back into the infrastructure. Step by step, each POP gradually shifted from Linux-based virtualization back to a native BSD environment.
This rolling migration strategy avoided any service interruption while keeping hardware utilization at healthy levels throughout the process.
Automation
If there is one lesson I learned while operating BoxyBSD over the years, it is this: never perform repetitive administrative work manually.
Every hypervisor, every network configuration, every storage pool, every monitoring component, and every virtual machine should be deployable automatically. Manual configuration always leads to configuration drift, inconsistencies, and eventually operational problems. Therefore, automation was one of the highest priorities during the migration.
Fortunately, the operating system itself required relatively little work. FreeBSD has always been well supported by Ansible, and my existing automation already handled the majority of system administration tasks. Installing software packages through pkg, configuring DNS resolvers, setting NTP servers, provisioning ZFS pools, configuring networking, managing users, deploying SSH keys, and performing operating system updates were already fully automated long before the migration started.
The virtualization layer itself, however, was entirely different.
Unlike Proxmox, there was no dedicated Ansible module available for Sylve. While Sylve already exposes a clean and well-designed REST API, interacting with that API directly through generic HTTP requests inside playbooks would quickly become difficult to maintain. Instead, I decided to write a dedicated Ansible module specifically for Sylve.
The module abstracts the complete lifecycle management of virtual machines. Creating instances, modifying hardware resources, attaching storage, managing networking, controlling power states, and removing guests can now all be described declaratively inside standard Ansible playbooks. This keeps infrastructure definitions clean while allowing the entire BoxyBSD platform to continue following the Infrastructure-as-Code philosophy that has been in place for years.
Thanks to Sylve's excellent API documentation, implementing the module was surprisingly straightforward. Most of the development effort was therefore spent integrating the new module into the existing provisioning workflows instead of reverse-engineering undocumented interfaces.
Today, deploying a new FreeBSD virtualization node requires little more than executing an Ansible playbook. Everything from the operating system installation to fully operational virtualization infrastructure happens automatically.
Monitoring
Running a free VPS hosting platform means that monitoring is far more than simply checking whether a server is online. Every free service eventually attracts abuse. Cryptocurrency mining, denial-of-service attacks, compromised virtual machines, spam campaigns, or users accidentally creating infinite loops all have one thing in common: they leave characteristic patterns inside system metrics.
Because of that, monitoring at BoxyBSD has always focused on predictive analysis instead of simple availability checks. CPU utilization, memory consumption, network throughput, disk latency, storage growth, ZFS health, and many other metrics are continuously collected and evaluated. Alerting does not only trigger after failures occur but also when systems show early warning signs that indicate future problems.
Moving back to bhyve introduced a new challenge. Existing monitoring exporters had been written around Proxmox's architecture and therefore no longer provided the required information. While Prometheus exporters exist for many virtualization platforms, none of them exposed exactly the metrics I needed from bhyve and Sylve.
As there aren't really any good bhyve exporters, I decided to build my own. The result became the bhyve Sylve Prometheus Exporter, which gathers virtualization metrics directly from the FreeBSD host and Sylve environment before exposing them in Prometheus format.
The exporter collects information about virtual machine states, CPU utilization, memory allocation, storage consumption, network traffic, and additional host statistics that integrate seamlessly into the existing Grafana dashboards and alerting rules. From the perspective of the central monitoring platform, very little changed despite replacing the complete virtualization stack underneath.
This allowed operational procedures to remain identical while providing even deeper insight into the BSD-based infrastructure than before.
Configuration Management Database (CMDB)
One component that fortunately required very little work was the Configuration Management Database.
Instead of relying on third-party CMDB software, BoxyBSD has always maintained its own lightweight implementation tightly integrated into the customer portal. Every virtual machine, physical server, IPv4 and IPv6 assignment, storage allocation, DNS configuration, monitoring target, and provisioning state is managed centrally through this backend.
Because the CMDB intentionally abstracts the underlying virtualization platform, very few modifications were necessary. Most backend functionality remained exactly the same. Only new connection strings, API endpoints, and virtualization-specific metadata had to be introduced to distinguish between Proxmox and Sylve managed systems.
This abstraction layer turned out to be one of the biggest advantages during the migration. Since higher-level services continued interacting only with the CMDB instead of directly with the hypervisors, almost no frontend or customer-facing code required any modification.
Keeping the Users Unaffected
Perhaps the most important design goal of the entire migration was ensuring that users never had to think about it.
Customers should not receive emails asking them to recreate virtual machines. They should not be asked to schedule maintenance windows, export backups, or reinstall operating systems. Their VPS should simply continue working exactly as before.
Achieving this required a migration strategy that operated entirely in the background.
Existing virtual machines remain on Proxmox until they naturally disappear. Whenever users delete VPS instances or stop using services, those resources become available again. Once enough virtual machines have disappeared from a Proxmox node, the remaining guests are live migrated onto another host. The now empty server is then reinstalled with FreeBSD and Sylve before rejoining the infrastructure as a native bhyve hypervisor.
At the same time, every newly created virtual machine is automatically provisioned on the new FreeBSD infrastructure. From the customer's perspective, nothing changes except that new instances are silently deployed onto bhyve instead of KVM.
This gradual migration strategy avoids unnecessary risk while continuously increasing the percentage of BSD-based infrastructure over time. Eventually, every Proxmox node will naturally become empty and can be converted without ever forcing existing users to migrate manually.
It is a surprisingly simple concept: instead of moving every virtual machine to the new platform, simply wait until the old platform slowly disappears on its own.
Missing Things
No infrastructure migration is complete without compromises, and moving back to FreeBSD with bhyve and Sylve is certainly no exception. While the BSD ecosystem has made tremendous progress over the past years, there are still areas where Linux-based virtualization platforms like Proxmox VE remain ahead. Some features are simply not available yet, while others require different architectural approaches. Fortunately, almost all of them have reasonable workarounds and, perhaps even more importantly, active development communities working towards closing these gaps.
Software Defined Networking (SDN)
One of the larger differences compared to my previous infrastructure is Software Defined Networking. Proxmox VE has invested heavily into SDN over the last years, allowing users to build sophisticated network topologies directly from the web interface. Technologies like VXLAN, EVPN, VLAN aware bridges, QinQ, and various routing models integrate nicely into the overall management experience.
Sylve currently focuses on the virtualization layer itself and therefore does not yet provide all of the SDN functionality that I previously relied on. Features such as QinQ support, EVPN integration, and some advanced overlay networking concepts are currently missing. While these are certainly desirable additions for larger multi-site deployments, they are not fundamental blockers for production environments.
In fact, FreeBSD has always been extremely flexible when it comes to networking. Long before SDN became a marketing buzzword, FreeBSD already offered an impressive toolbox for building complex virtual networks. Instead of relying on integrated management interfaces, these features can be implemented directly using native operating system components. I already demonstrated this years ago while running CBSD by building a stretched Layer 2 network between multiple bhyve hosts, which is documented in my article Creating a Stretched Layer 2 Network for VMs in bhyve on FreeBSD.
Although a built-in SDN management layer would certainly improve the user experience, the underlying capabilities are already there. It is therefore mostly a question of management tooling rather than technical limitations of FreeBSD itself.
Backups
Another area where Proxmox provides an outstanding user experience is the integration with Proxmox Backup Server (PBS). The combination of incremental backups, global deduplication, client-side encryption, scheduled backup jobs, and one-click file or virtual machine restores is difficult to beat. Especially for service providers, PBS significantly simplifies backup management while keeping storage requirements remarkably low.
Leaving Proxmox also means leaving this excellent integration behind.
Instead of replacing PBS with another backup appliance, I decided to move back to one of FreeBSD's greatest strengths: ZFS.
Every virtual machine is stored on encrypted ZFS datasets which are protected through regular snapshots. Those snapshots are then replicated to offsite locations using ZFS send and receive. This approach has several advantages. Replication is incremental, highly efficient, cryptographically secure through native encryption, and completely independent of any third-party backup software. Since ZFS replication operates on the filesystem level, complete datasets can be restored quickly while maintaining all filesystem metadata.
Although it lacks the polished web interface of Proxmox Backup Server, the underlying technology is extremely robust and has proven itself for many years in enterprise storage environments. It also aligns perfectly with the philosophy of keeping the infrastructure as close to native FreeBSD technologies as possible.
Live Migrations
Without any doubt, the largest missing feature today is live migration.
During my years on Proxmox VE, live migrations became so normal that I almost forgot how revolutionary they actually are. Need to install operating system updates? Simply migrate the virtual machines elsewhere. Hardware maintenance? Move the workloads to another node. Storage replacement? Live migrate everything away first. Users never notice anything.
Unfortunately, this convenience does not yet exist in bhyve.
The inability to move running virtual machines between physical hosts is currently the biggest operational drawback of the BSD virtualization ecosystem. Maintenance windows once again require shutting down virtual machines before rebooting the host, something many administrators haven't had to think about for years. It almost feels like stepping back into the late 1990s, when scheduled downtimes were simply accepted as part of operating a server.
Fortunately, the bhyve community has been discussing live migration support for quite some time, and I am optimistic that this capability will eventually become reality. Once it arrives, one of the largest remaining gaps between bhyve and commercial virtualization platforms will disappear almost overnight.
Dynamic Resource Scheduling
Readers who have followed my previous articles might already have guessed the next missing feature.
Dynamic workload placement has become an essential part of operating larger virtualization clusters. During my Proxmox years I developed ProxLB, bringing VMware DRS-like resource scheduling to Proxmox by continuously analyzing cluster utilization and automatically balancing virtual machines across available hosts.
Without live migrations, however, continuously balancing running workloads simply does not make sense.
That does not mean intelligent placement disappears entirely.
Instead, I am currently working on a new scheduler specifically designed for Sylve. Rather than continuously moving running guests, it focuses on making the best possible placement decision at deployment time. When a new virtual machine is created, the scheduler evaluates available CPU capacity, memory usage, storage utilization, and other metrics before selecting the most suitable host. While this is technically not a DRS implementation, it still prevents clusters from becoming unbalanced over time.
Should live migrations eventually become available in bhyve, extending this placement utility into a full dynamic resource scheduler will become a relatively straightforward evolution.
Conclusion
After several years running BoxyBSD on Proxmox VE, moving back to FreeBSD initially felt like taking a step into the unknown. Looking back now, I would describe it differently. It feels much more like coming home.
The BSD ecosystem has evolved enormously during the last few years. Projects like Sylve demonstrate that modern virtualization on FreeBSD is no longer just a niche for enthusiasts but a serious platform capable of operating production workloads at scale. While some features are still missing compared to more mature Linux ecosystems, the foundation is incredibly solid and, perhaps even more importantly, built around technologies that have always made BSD attractive: simplicity, transparency, stability, and clean engineering.
What impressed me most throughout this migration was not only the technical quality of Sylve itself, but also the speed at which it continues to evolve. New functionality is added continuously, documentation is improving rapidly, and the community surrounding the project is highly responsive. It reminds me very much of the early days of Proxmox before it became the platform we know today.
Of course, there is still work ahead. Live migrations remain the biggest missing feature, SDN capabilities will continue to mature, and the surrounding tooling ecosystem is still significantly smaller than its Linux counterpart. But none of these challenges feel insurmountable. On the contrary, they represent exciting opportunities for the BSD community to innovate and build solutions tailored specifically to FreeBSD instead of simply copying existing Linux concepts.
For me personally, this migration also marks a philosophical shift. BoxyBSD was always created to promote BSD operating systems, demonstrate their capabilities, and lower the entry barrier for newcomers. Running the infrastructure itself on Linux always felt slightly contradictory to that mission. Today, the platform once again runs on the technologies it advocates every single day.
Will every virtualization environment immediately switch from Proxmox to Sylve? Certainly not, and they probably shouldn't. Different environments have different requirements, and Proxmox remains an outstanding virtualization platform. But for organizations, enthusiasts, and service providers that already embrace FreeBSD, Sylve has become an incredibly compelling solution that deserves serious consideration.
Most importantly, Sylve proves something that many people have questioned for years: modern virtualization on FreeBSD is alive, actively developed, and moving forward at an impressive pace. I have no doubt that the project will continue closing the remaining feature gaps over the coming years. Judging by the progress already made, I believe the future of bhyve has never looked brighter.
For BoxyBSD, the journey has come full circle. It started on FreeBSD with bhyve many years ago, temporarily left to build a more automated platform on Proxmox, and now returns stronger than ever. This time, however, it returns with an ecosystem of automation, monitoring, tooling, and experience that simply did not exist back then. And thanks to projects like Sylve, I am convinced that this time, it is here to stay.