One of the recurring topics when discussing Proxmox VE as an alternative to VMware is shared storage. While VMware has offered VMFS for many years, providing a clustered filesystem that allows multiple ESXi hosts to access the same shared storage concurrently, Proxmox VE has traditionally approached this differently.

There are several ways to provide shared storage in Proxmox VE, including iSCSI, Fibre Channel (FC), NVMe over Fabrics (NVMe-oF), shared LVM, and distributed storage solutions such as Ceph. However, each approach comes with its own advantages and limitations.

In my recent article Proxmox VE Shared Storage: LVM Thin, MeshXFS, CetraFS, I discussed the current storage situation and looked at different approaches to closing the gap between Proxmox VE and what administrators are used to from VMware environments.

Warning: CetraFS is still in a very early state and should not yet be used in production setups!

One particularly interesting project in this context is CetraFS. It takes a different approach compared to traditional LVM-based storage backends by providing an independent clustered filesystem that can be mounted by multiple nodes simultaneously.

CetraFS is designed to provide shared-disk filesystem capabilities, allowing virtual machine disk images to reside directly on a shared block device. Its architecture includes coordinated allocation, per-node write-ahead journals, checksums, sparse-file support, and atomic SCSI COMPARE AND WRITE operations for coordinating access between nodes.

The idea is to provide functionality comparable to what administrators know from VMFS, while using existing shared block storage infrastructure such as iSCSI or Fibre Channel.

In addition to working on the project, I recently contributed a native Proxmox VE storage integration, including an additional web UI component. This makes it possible to manage CetraFS directly through the familiar Proxmox VE interface instead of relying exclusively on CLI commands.

In this article, we will look at two different ways to install CetraFS on Proxmox VE:


  • Installing CetraFS from source to get the newest development versions.
  • Installing CetraFS through my Debian repository using prebuilt packages.

Prerequisites

Before getting started, there are a few things to consider. CetraFS is designed for shared block storage, meaning that all participating Proxmox VE nodes need access to the same underlying storage device.

Typical storage protocols include:


  • iSCSI
  • Fibre Channel (FC)
  • NVMe over Fabrics (NVMe-oF)

The underlying storage hardware must support the atomic SCSI COMPARE AND WRITE operations required for coordination. Make sure that your storage array, multipath configuration, and complete I/O path support the required operations.

Furthermore, the required packages and kernel module need to be installed on every Proxmox VE node that is expected to access the shared filesystem.

For this guide, I am using Proxmox VE 9 on Debian Trixie, but before starting please take note, that CetraFS is still in a very early state and should not yet be used in production setups!

Method 1: Installing CetraFS from Source

The first installation method is to build CetraFS directly from its source code. This approach is particularly useful if you want to test the newest development versions, work with specific branches, or contribute to the project.

First, create a working directory and clone the Git repository: 


mkdir -p /opt/cetra
cd /opt/cetra
git clone https://github.com/cetrafs/cetrafs.git
cd cetrafs

Next, configure the kernel module. In this example, we are setting the stale timeout to 120 seconds: 

echo "options cetrafs stale_secs=120" > /etc/modprobe.d/cetrafs.conf

Now we can build the project and its components: 


make
make mod
make userspace

Once the build has completed successfully, prepare the source tree for DKMS. For this example, we are using version 0.8.2: 


cp -r . /usr/src/cetrafs-0.8.2

Register the module with DKMS and build it for the currently installed kernel: 


dkms add -m cetrafs -v 0.8.2
dkms build -m cetrafs -v 0.8.2
dkms install -m cetrafs -v 0.8.2

If you need to force a reinstall, for example after modifying an existing build, you can use: 


dkms install -m cetrafs -v 0.8.2 --force

Finally, load the kernel module: 


modprobe cetrafs

You can verify that the module is loaded with: 


lsmod | grep cetrafs

Creating and mounting a CetraFS filesystem


With the kernel module loaded, we can initialize a block device and mount the filesystem.

For this example, I am using /dev/sdb. Make sure to replace this device with the correct shared block device in your environment.

Warning: Formatting a device destroys existing data. Only execute this command on a dedicated, empty block device. 


mkfs.cetra /dev/sdb -f

After formatting, create a mount point and mount the filesystem: 


mkdir -p /mnt/test
mount /dev/sdb /mnt/test

You should now have a mounted CetraFS filesystem and can start testing basic filesystem operations.

Method 2: Installing CetraFS through the Debian Repository

If you do not want to compile CetraFS manually, there is a more convenient alternative. I have made the project available through my Debian package repository at packages.gyptazy.com.

This approach simplifies the installation process significantly. Instead of manually compiling the kernel module and installing the userspace tools, we can use the regular APT package management workflow.

First, install the required dependencies, including the matching Proxmox VE kernel headers: 


apt-get -y install curl sudo pve-headers-$(uname -r)

Next, download and install the repository signing key: 


curl https://git.gyptazy.com/api/packages/gyptazy/debian/repository.key \
 -o /etc/apt/keyrings/gyptazy.asc

Add the repository to your APT sources: 


echo "deb [signed-by=/etc/apt/keyrings/gyptazy.asc] https://packages.gyptazy.com/api/packages/gyptazy/debian trixie main" \
 | sudo tee -a /etc/apt/sources.list.d/gyptazy.list

Update the package lists: 


apt-get update

Now install the CetraFS kernel module and userspace tools: 


apt-get -y install cetrafs-dkms cetrafs-tools

The DKMS package automatically builds and installs the kernel module for the installed kernel, provided that the matching kernel headers are available.

Installing the Proxmox VE integration

Besides the filesystem itself, I have also recently contributed a native Proxmox VE storage integration. This includes the storage backend and an additional UI component, allowing administrators to manage CetraFS directly through the Proxmox VE web interface.

To install the additional integration packages, simply execute: 


apt-get -y install pve-storage-cetrafs pve-storage-cetrafs-ui

This installs the Proxmox VE storage plugin and its corresponding web UI integration.

Once the packages are installed, the additional CetraFS integration becomes available within the native Proxmox VE management interface.

This is an important step towards making CetraFS easier to deploy and operate in everyday Proxmox VE environments. Instead of managing everything through shell commands, administrators can use the same familiar interface they already know from other Proxmox VE storage backends.

Creating a filesystem

Warning: This destroys everything on your device!
As with the source-based installation, we need to initialize our shared block device before using it: 


mkfs.cetra /dev/sdb -f

After the filesystem has been initialized, it can be configured and managed through the Proxmox VE integration.

Proxmox VE Web UI Integration

One of the things I wanted to improve was the overall usability of CetraFS within Proxmox VE. Having a working clustered filesystem is one thing, but making it easy to configure and manage is equally important.

That is why I recently brought in a native Proxmox VE UI addition for CetraFS.

The integration provides a familiar way to work with CetraFS directly inside the Proxmox VE web interface, rather than having to perform every storage-related operation manually through the command line.

The implementation consists of two packages:


  • pve-storage-cetrafs - The Proxmox VE storage backend.
  • pve-storage-cetrafs-ui - The additional web interface integration.

This separation also makes it possible to install the filesystem independently from the Proxmox VE-specific integration.

For further development and implementation details, take a look at the project repository and the corresponding integration work:



Important Considerations

Although CetraFS offers an interesting approach to shared storage on Proxmox VE, there are a few important considerations before using it in production environments.

First of all, CetraFS is still a young project and has not reached version 1.0 yet. It should therefore not be considered a drop-in replacement for mature and extensively validated solutions such as VMware VMFS.

Second, the underlying storage hardware must support the required atomic COMPARE AND WRITE operations. This is particularly important when working with shared SAN storage and multipath configurations.

Furthermore, correct fencing and recovery procedures are essential. A clustered filesystem must ensure that nodes which have lost access or become isolated cannot continue writing to shared storage in an unsafe manner.

Before using CetraFS for critical workloads, I strongly recommend testing the following scenarios:


  • Concurrent read and write access from multiple nodes.
  • Node failures and recovery.
  • Storage path failures and multipath behavior.
  • Filesystem consistency after unexpected shutdowns.
  • Virtual machine migrations between cluster nodes.
  • Recovery and fencing behavior under network partitions.
  • Performance under representative virtualization workloads.

As always, make sure you have independent backups and a tested recovery strategy.

Conclusion

CetraFS is an interesting addition to the Proxmox VE storage ecosystem. It explores a different approach to shared storage by providing a clustered filesystem on top of shared block devices, allowing multiple nodes to access the same filesystem concurrently.

For administrators coming from VMware environments, this is particularly interesting because it addresses one of the familiar VMFS use cases without requiring a completely different distributed storage architecture.

With the availability of Debian packages, the installation process becomes much easier. Administrators can choose between building directly from source to access the newest development versions or using the Debian repository for a more convenient package-based installation.

The additional Proxmox VE storage plugin and native web UI integration are another step towards making CetraFS easier to use and manage within existing Proxmox VE clusters.

There is still development and validation work ahead, especially regarding performance, recovery, and production readiness. Nevertheless, I am looking forward to seeing how CetraFS evolves and what possibilities it will bring to the Proxmox VE storage ecosystem.

For more information, visit the CetraFS GitHub repository or the official CetraFS website.