CBSD on FreeBSD – bhyve Virtualization with VXLAN and Stretched Layer-2 Networking

In today's globalized IT landscape the term "cloud" dominates conversations about infrastructure, applications, and deployment strategies. Public cloud providers promise scalability, flexibility, and resilience but yet many organizations still operate their own infrastructure for reasons of control, cost, and compliance.

In these environments, FreeBSD continues to play an important role as a robust, secure, and versatile operating system.

CBSD as a bhyve Management Layer on FreeBSD

One of the powerful tools in the FreeBSD ecosystem is CBSD (github). CBSD acts as a management layer that simplifies the handling of FreeBSD jails, bhyve virtual machines, and other system resources. Instead of manually working through complex configuration steps, administrators can rely on CBSD’s unified command-line and TUI interfaces to create, configure, and maintain VMs and containers with ease. But there is also a WebUI bundled within ClonOS which makes use of cbsd. In particular, CBSD makes bhyve—the native FreeBSD hypervisor—far more accessible, allowing administrators to spin up virtual machines quickly and efficiently.

While virtualization is an essential building block, networking remains equally critical. In an age where businesses and applications often span multiple data centers and geographical locations, the need for seamless connectivity grows. Traditional layer-3 routing is not always sufficient, especially when workloads must operate as though they are on the same local subnet despite being physically distributed across continents.

This is where VXLAN (Virtual Extensible LAN) technology comes into play. By creating VTEPs (VXLAN Tunnel Endpoints) on participating hosts, administrators can establish stretched layer-2 networks that extend across long distances. Such VXLAN-based overlays make it possible to connect bhyve VMs—even when they run in different data centers or countries—as if they were sitting in the same rack.

Combining CBSD's streamlined VM lifecycle management with VXLAN-based networking opens the door to building flexible, distributed infrastructures without the complexity often associated with large-scale systems. Whether you are hosting applications in Europe and Asia or simply connecting two regional sites, this approach provides a foundation for reliable, scalable, and geographically stretched virtualized environments.

This blog post introduces CBSD on FreeBSD, explains its role in managing bhyve-based VMs, and highlights how VXLAN technology helps create resilient, long-distance layer-2 networks. Together, these tools enable administrators to build modern, cloud-like infrastructures while retaining the stability and control of FreeBSD.

Btw, hopefully this post makes some of my friends in the BSD community happy to see me back there ;)

Why Traditional Layer-3 Networking Becomes a Problem

Running virtual machines inside a single data center is usually straightforward. They share the same local network, which makes communication easy and predictable. Problems start to appear as soon as workloads are spread across different sites or even across countries. At that point networking quickly turns into one of the hardest parts of the design.

Designing Layer-2 Overlays Across Multiple Data Centers

In a distributed setup virtual machines often end up in different subnets. That means traffic has to be routed between sites, which adds complexity and sometimes forces the use of NAT. Applications that expect to run on the same LAN suddenly behave differently or stop working altogether. To connect sites administrators usually rely on site-to-site VPNs such as IPsec, WireGuard, or OpenVPN. These work well for encrypted connections at the IP layer, but they do not provide a shared layer two domain. Protocols that need broadcast, multicast, or direct adjacency across an Ethernet segment are not supported in this model. As a result, clustering software, distributed storage, and other systems that depend on direct neighbor discovery often run into limitations.

There is also the issue of performance. VPNs usually add latency because all traffic is encapsulated and routed through tunnels. With every additional site the amount of configuration grows, since new tunnels and routing policies must be set up and maintained. This operational overhead can make scaling across several locations difficult.

VXLAN Fundamentals for Stretched Layer-2 Networks

The goal in such a scenario is to make workloads behave as if they were still on the same local network, no matter where they are physically located. Virtual machines in different data centers should be able to communicate directly, exchange ARP requests, use broadcast, and rely on the same protocols they would use in a single site. What administrators want is a stretched layer two network that connects all locations into one logical LAN.

Conclusion: Cloud-Like Networking with FreeBSD, CBSD, and VXLAN

To achieve this, the network must provide an overlay that hides the complexity of the underlying transport. VXLAN technology is well suited for this purpose. By creating virtual tunnel endpoints on each host, VXLAN allows administrators to extend layer two networks across IP links. With this approach, a machine in Europe and another in Asia can operate in the same broadcast domain. From the perspective of the workloads, distance no longer matters (at least as it's not tied to latency critical applications).