Proxmox VE 9 Storage Benchmark: Shared LVM Thin/Thick vs CetraFS

Shared storage remains one of the recurring topics when evaluating Proxmox VE as an alternative to VMware. In particular, administrators who are used to VMFS often want to keep their existing SAN infrastructure while gaining the flexibility of a clustered storage backend.

As discussed in my overview of the Proxmox VE shared-storage landscape in late 2026, several approaches are emerging to address different parts of this challenge. This benchmark takes a closer look at three of them: shared LVM Thick, shared LVM Thin using my custom Proxmox VE storage plugin, and CetraFS.

The goal is to compare their measured performance under two conditions: an isolated single-VM workload and a concurrent workload generated by three VMs running across three hypervisor nodes. The tests cover sequential throughput, random I/O, mixed workloads, and read latency. The results are not intended to establish a universal ranking for every environment, but to show how these configurations behaved in this particular test setup.

Storage Backends Under Test

The three backends take different architectural approaches and should not be treated as interchangeable implementations of the same storage model.

• Shared LVM Thick: Traditional LVM on shared block storage, such as an iSCSI or Fibre Channel LUN. The logical volumes are thick-provisioned. Proxmox VE 9 also introduced snapshot support for shared LVM through volume chains.
• Shared LVM Thin: This benchmark uses my custom Shared LVM Thin storage plugin. It is not the stock Proxmox VE lvmthin backend configured as shared storage. The plugin uses a dedicated thin pool per guest within a shared volume group, with the guest's pool activated exclusively on the node currently running that guest.
• CetraFS: A developing shared-disk clustered filesystem designed to allow multiple nodes to access the same shared block device concurrently. It uses its own filesystem implementation and requires compatible storage hardware, including support for atomic SCSI COMPARE AND WRITE operations.

For more background on the architecture and current maturity of these approaches, see the shared-storage overview and my guide to installing CetraFS on Proxmox VE 9.

Test Setup and Methodology

All benchmarks were executed inside virtual machines using fio 3.39, the libaio I/O engine, and direct=1 to bypass the guest page cache. Each test used a 10-second ramp-up followed by a measured runtime of approximately 120 to 180 seconds.

The following workloads were used:

• Sequential read: 1 MiB block size, queue depth 16.
• Sequential write: 1 MiB block size, queue depth 16.
• Random read: 4 KiB block size, queue depth 32.
• Random write: 4 KiB block size, queue depth 32.
• Mixed random read/write: 4 KiB block size, queue depth 32, 70% reads and 30% writes.
• Read latency: 4 KiB random reads, queue depth 1.

Two scenarios were evaluated:

• Single VM: One VM executed the fio workload without competing benchmark VMs.
• Multi VM: Three VMs ran identical workloads concurrently. They were distributed across three hypervisor nodes and accessed the same shared storage.

In the multi-VM scenario, the results shown for each backend represent the workload observed by the reported fio job while two additional VMs generated concurrent I/O. This is intended to approximate a busier cluster than the isolated single-VM test.

Important: The results below describe this test environment only. Hardware details such as the storage array model, drive type, controller configuration, SAN protocol, multipath configuration, network topology, cache policy, VM configuration, and exact fio job files should be documented alongside the results to make the benchmark independently reproducible. They are not specified in the benchmark data supplied for this article, so no assumptions about them are made here.

Single-VM Benchmark Results

The single-VM results show the performance of each backend without competing benchmark VMs. LVM Thick leads in most random and mixed workloads, while LVM Thin produces the highest sequential-read throughput. CetraFS is behind the two LVM configurations in most throughput measurements, although its p99 latency in the queue-depth-one read test is marginally lower than LVM Thin's.

What stands out?

LVM Thin delivered approximately 1.76 times the sequential-read throughput of LVM Thick in this run. This is a notable result, but it should not be attributed to thin provisioning itself without further investigation. Allocation layout, test-data placement, cache behaviour below the guest, and the state of the underlying storage can all affect sequential-read measurements. Repeated runs and verification that the test data is fully allocated and read from the intended storage path would help establish whether this difference is consistent.

LVM Thick performed best in sequential writes, random reads, random writes, and both sides of the mixed workload. Its random-read result was particularly strong at 56,700 IOPS, compared with 34,700 IOPS for LVM Thin and 33,700 IOPS for CetraFS.

The queue-depth-one latency results are closer. LVM Thick recorded the lowest median latency at 223 µs, while CetraFS had the lowest p99 value at 594 µs. This illustrates why throughput, median latency, and tail latency should be considered separately rather than condensed into a single winner.

Multi-VM Benchmark Results

The multi-VM scenario is particularly relevant for shared storage: three VMs on three hypervisor nodes generated concurrent I/O against the same backend. All three configurations experienced a substantial performance reduction compared with the isolated tests, but the degree of degradation differed.

*CetraFS sequential-write result: see the test anomaly above. The extended runtime makes this result unsuitable for a direct like-for-like comparison.

What stands out?

LVM Thick retained a clear lead in random reads and writes. It delivered 13,800 random-read IOPS, while LVM Thin reached 2,028 IOPS and CetraFS reached 1,507 IOPS. The latency results also show a large gap: the reported p50 read latency was approximately 570 µs for LVM Thick, compared with 4,490 µs for LVM Thin and 9,503 µs for CetraFS.

The mixed workload results are more closely matched between the two LVM configurations. LVM Thick reached 952 read IOPS and 408 write IOPS, while LVM Thin reached 944 read IOPS and 405 write IOPS. CetraFS recorded lower values in both directions.

CetraFS achieved 338 MiB/s in sequential reads, relatively close to LVM Thick at 371 MiB/s. However, its random I/O and latency results were considerably weaker in this particular concurrent test. The sequential-write anomaly also means that result needs to be repeated before drawing conclusions.

Comparing Single-VM and Multi-VM Behaviour

The difference between isolated and concurrent workloads is one of the most important findings in this benchmark. For random reads, LVM Thick decreased from 56,700 to 13,800 IOPS, a reduction of approximately 76%. LVM Thin decreased from 34,700 to 2,028 IOPS, while CetraFS decreased from 33,700 to 1,507 IOPS.

In percentage terms, the LVM Thin and CetraFS random-read results fell by approximately 94% and 96%, respectively. These figures describe the observed change between the two test scenarios; they do not, on their own, identify the underlying cause. Further profiling would be required to determine whether the bottleneck comes from filesystem or volume-management overhead, locking, queueing, the shared storage target, or another part of the I/O path.

The latency measurements reinforce the throughput results. Under concurrent load, the reported p99 read latency increased to approximately 3.0 ms for LVM Thick, 64.8 ms for LVM Thin, and 95.9 ms for CetraFS. For latency-sensitive virtual machines, this difference can matter as much as peak throughput.

Interpreting the Results

Why was LVM Thin faster in sequential reads?

The single-VM sequential-read result for LVM Thin is significantly higher than the other two results. The benchmark data alone cannot establish why. Potential factors worth checking include the physical layout of the test data, thin-volume allocation patterns, cache behaviour in the storage array or lower layers, and whether all test blocks were allocated and read consistently. A repeat test with freshly prepared volumes, controlled cache settings, and verification at the block-device layer would help isolate the cause.

Why did performance change under concurrent I/O?

The multi-VM scenario adds contention that is absent from the single-VM test. Multiple guests issue requests at the same time, and those requests share the same storage path. The observed results may therefore reflect a combination of backend behaviour and limits elsewhere in the stack, including storage-controller queue depth, SAN connectivity, multipath configuration, host resources, and I/O scheduling.

It would be premature to attribute the CetraFS or LVM Thin degradation to one specific metadata or locking mechanism based on fio output alone. Tracing, storage-side telemetry, per-node statistics, and repeated tests would be needed to identify the bottleneck.

What does the CetraFS result tell us?

CetraFS delivered reasonable sequential-read throughput in the multi-VM test, but its random I/O throughput and reported latency were significantly weaker in this run. The sequential-write test also showed an unusually long runtime and should be repeated. These measurements are useful signals for further development and testing, but they are not sufficient to make a general statement about CetraFS performance across different hardware, workloads, or versions.

CetraFS is still a young project and should not be treated as a production-ready drop-in replacement for mature clustered filesystems. Its suitability needs to be assessed through more than performance tests: concurrent access, fencing, crash recovery, multipath failures, node isolation, and data integrity all need to be validated. See the CetraFS installation guide and its operational considerations for further details.

Benchmark Limitations and Next Steps

This benchmark provides an initial comparison, not a comprehensive storage qualification. Several additions would make a future revision more reproducible and useful for capacity planning:

• Publish the exact server, CPU, memory, HBA/NIC, storage-array, drive, and controller configuration.
• Document the storage protocol, multipath policy, queue settings, and relevant host and guest kernel versions.
• Publish the complete fio job files, including runtime, ramp time, file size, I/O pattern, number of jobs, and reporting options.
• Repeat each test several times and report run-to-run variation, not only a single result.
• Ensure that test volumes and data are prepared consistently, and document cache policies and whether reads can be served from cache.
• Measure the storage array and host I/O paths during the tests to identify queue saturation, controller limits, or other bottlenecks.
• Run longer mixed workloads and include additional concurrency levels to understand how performance scales beyond three VMs.
• For clustered filesystems, add dedicated failure, fencing, recovery, and data-integrity tests. Performance alone is not enough to qualify a shared-disk filesystem.

The distinction between average latency and percentile latency should also be retained when publishing future results. The p50 value describes the median completion latency, while p99 highlights the slower tail of the I/O distribution. A backend can have good average throughput and still produce latency spikes that are visible to applications.

Conclusion

In this test, shared LVM Thick delivered the most consistent overall performance, particularly in random I/O and mixed read/write workloads. It also showed substantially better latency than the other two configurations in the concurrent multi-VM scenario. For workloads that generate frequent random I/O or run several busy VMs against the same shared storage, this makes LVM Thick the strongest performer among the three configurations tested here.

Shared LVM Thin showed an interesting advantage in single-VM sequential reads and remained close to LVM Thick in the multi-VM mixed workload. However, its random-read performance and latency degraded substantially in the concurrent test. Its results should be considered in the context of the custom plugin architecture and the specific test environment; this is not a benchmark of Proxmox VE's stock local lvmthin backend.

CetraFS produced competitive sequential-read throughput in the multi-VM test, but its random I/O and latency results need improvement or further investigation. The anomalous sequential-write run should be repeated. More importantly, performance is only one part of evaluating a clustered filesystem: fencing, recovery, consistency, and failure handling are essential requirements before considering critical workloads.

These results should therefore be read as a snapshot of three implementations in one test environment, not as a universal verdict. Shared LVM Thick is the clear performance reference in these measurements, while Shared LVM Thin and CetraFS represent different approaches with their own operational and architectural trade-offs. As the Proxmox VE storage ecosystem continues to evolve, repeating these tests with newer releases and publishing a fully reproducible test setup will be just as important as the numbers themselves.

For the broader architectural context, take a look at my article on the Proxmox VE shared-storage situation in late 2026. For implementation details, see Shared LVM Thin provisioning on shared block storage and Installing CetraFS on Proxmox VE 9.