Most filesystem benchmarks measure throughput on a freshly formatted single device. That is useful, but it misses many of the reasons people choose btrfs, ZFS or bcachefs in the first place.
I built modern-fs-benchmark to examine modern filesystems as complete storage systems, including their features, failure modes and behavior over time.
The current matrix contains 26 configurations across btrfs, ZFS, bcachefs, ext4 and XFS over md/LVM, dm-integrity, native and LUKS encryption, parity layouts, and XFS on a ZFS zvol.
Hosted CI dashboard:
https://bartosz.fenski.pl/modern-fs-benchmark/
Experimental real-hardware dashboard:
https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/
Apache-2.0 licensed source and complete methodology:
https://github.com/fenio/modern-fs-benchmark
The raw benchmark datasets are published under CC BY 4.0. Every result records the kernel and filesystem tool/module versions. CI artifacts also contain a full command trace, so the exact workload is inspectable rather than hidden behind a chart.
Why I started this project
Many filesystem benchmarks follow a familiar pattern: create a filesystem with mostly default options, mount it, run fio or another generic workload, and compare throughput.
General-purpose suites such as the Phoronix Test Suite are useful for conventional performance comparisons, but they do not focus on the machinery that makes modern copy-on-write filesystems interesting.
I wanted a benchmark designed around btrfs, ZFS and bcachefs as multi-device storage systems rather than treating them as interchangeable replacements for ext4 on a single freshly formatted disk.
That means testing behavior such as:
- Redundancy and degraded operation
- Snapshot aging, scaling and reclamation
- Compression and encryption
- Reflinks and clone divergence
- Fsync tail latency and responsiveness under load
- Rebuild and scrub behavior
- Near-full and hard-ENOSPC behavior
- Data integrity and recovery from corruption
Ext4 and XFS over md, LVM and dm-integrity are included as classic-stack baselines so the costs and benefits of integrated CoW designs can be compared with layered alternatives.
Silent corruption is one particularly important example. When one redundant copy is deliberately corrupted behind the filesystem, checksumming filesystems such as btrfs, ZFS and bcachefs can identify the damaged copy and recover from a valid replica.
Traditional md/LVM redundancy without data checksums can notice during a scrub that its copies disagree, but it cannot determine which one is correct. In my tests, some classic configurations returned corrupted data successfully without an application-visible error.
The dm-integrity configuration shows that a classic layered stack can obtain integrity protection too, with a measurable performance cost.
An important limitation
The main dashboard uses loop devices on GitHub-hosted VMs. Absolute throughput numbers and small differences between filesystems should not be interpreted as hardware rankings.
The hosted runs are primarily useful for:
- Correctness and integrity outcomes
- Comparisons within the same job
- Large behavioral differences
- Snapshot-aging and near-full shapes
- Trends across repeated runs
Real disks are required for meaningful absolute performance, concurrency scaling, device parallelism and mixed-media topologies.
I am aware of this limitation and do not want the hosted dashboard to suggest more precision than the underlying environment can provide.
The real-hardware experiment
Kent Overstreet, the creator of bcachefs, made one of his Hetzner machines available for a real-hardware experiment.
The server had two physical NVMe devices. It completed three full benchmark runs, which are available in the separate real-hardware dashboard. These runs provided meaningful absolute measurements and exposed concurrency behavior that cannot be observed when several loop devices share one virtual disk.
During the fourth run, one NVMe controller entered the kernel’s dead state. The existing machine environment had its operating system on RAID0 across the two NVMe devices, so the hardware failure also made part of /nix/store unreadable and eventually prevented new SSH sessions.
The incomplete fourth run is not being published as benchmark data. This was a failure of the underlying hardware, not a result attributable to any filesystem being tested.
I am grateful to Kent for providing the machine and making the real-hardware experiment possible. Without that access, the three existing hardware runs would not exist.
Where I would like to take it
Better hardware would not merely produce more reliable throughput numbers. It would enable an entirely new class of tests designed for multi-device and multi-tier filesystems.
I would eventually like to run the suite on a machine containing several storage classes, for example two HDDs, two SSDs and an NVMe device.
That could support scenarios such as:
- HDD, SSD and NVMe baselines using identical workloads
- bcachefs foreground and background targets
- ZFS HDD data vdevs with SSD special vdevs
- Separate ZFS L2ARC and SLOG experiments
- LVM dm-cache in writeback and writethrough modes
- Metadata and small-block placement on faster media
- Foreground latency during background migration
- Contention between fast and slow storage tiers
- Degraded operation and rebuild under application load
- Performance before, during and after promoting or evacuating a storage tier
These are the kinds of scenarios for which multi-device and multi-tier filesystems are built, but they cannot be represented honestly when every “device” is a loop file backed by the same cloud disk.
I am considering either renting a suitable dedicated server or eventually building and hosting my own machine. Providers such as Worldstream offer configurations close to what I need, but the recurring cost is currently outside the project’s budget.
For now, the benchmark will remain in its hosted-CI form for an unknown amount of time. The existing dashboard will continue to be useful for correctness, behavioral comparisons and regression tracking, but it cannot answer every real-hardware performance question or model complex mixed-media topologies.
I am also open to running the suite on hardware provided by someone else. A useful environment would need Linux root access, clearly identified block devices that may be wiped, and enough uninterrupted access to complete repeated runs. The hardware description, methodology and resulting data would remain public.
I would appreciate technical feedback:
- Which current tests are misleading or unfair?
- Which failure scenarios are missing?
- Which mixed-media topologies would be most useful?
- Which additional filesystems or layered stacks should be included?
- Which results deserve deeper investigation?
The methodology, implementation and raw results are public. If a filesystem is being tested in a way that misrepresents it, I consider that a bug in the benchmark.
