Help understanding options for expanding ZFS storage server

I essentially came here to say everything @ThisMightBeAFish said.

It is generally bad practice to layer file system tools on top of other file system tools, as they can wind up fighting with each other and having unintended consequences, up to and including data loss in some rare occasions.

I also agree that RAIDz was probably not the best choice, but for a four drive pool it becomes pretty difficult to stomach the capacity efficiency loss of the better alternatives (like striped mirrors or RAIDz2 or RAIDz3…

But there is one thing he didn’t touch upon.

Traditionally you had only two ways to expand an existing ZFS pool (without breaking it up and starting over). Those were:

• Replace smaller drives with larger drives one by one, resilvering each, and once the last drive is resilvered if autoexpand=on is set, the pool grows to fill the space.

–OR–

• Adding an additional VDEV that will then be striped, preferably of the same configuration as the first, but ZFS won’t stop you from mixing and matching. Mixing and matching VDEV configurations is generally not considered a good idea though.

(Worth nothing here that while striping for single drives - so called RAID 0 - has a bad reputation, and deservedly so, striping in this context, across stable redundant VDEV’s is well tested, and an intended way to use ZFS and as such is not a problem, as long as you plan your underlying redundancies correctly.)

There has been an update to this though.

Starting with OpenZFS 2.3 RAIDz expansion became possible.

You can now add drives to an existing RAIDz, RAIDz2 or RAIDz3 vdev.

–BUT–

When doing so, you cannot change the redundancy level. So, you cannot turn an existing four drive RAIDz1 vdev into a RAIDz2 or RAIDz3 vdev of any size. RAIDz expansion must retain the same level of redundancy as the original vdev being expanded.

What this means for you.

You can add a second vdev, and wind up with a pool consisting of two separate four drive RAIDz vdevs, but - honestly - in 2026, single drive parity is really no longer considered sufficient, especially as drive counts grow.

I know it is painful to hear, but the best path forward is to admit that you made a mistake with RAIDz1 back when you created the pool (no criticism intended here, we all learned and made mistakes at some point) and start over.

This means that - unless you want to buy all new drives, which is expensive, especially these days - you need a way to back up your data, while you break up your pool and re-form it.

Honestly though, you should already have this. Redundancy is not a backup, and you really ought to have backups of your data, especially if it is important and isn’t easily replaceable.

But once you get over that hurdle, how should you set this up?

Back in the day performance was very sensitive to the combination of drive counts at each redundancy level based on partition level and stripe size as follows:

sub.mesa wrote:
As i understand, the performance issues with 4K disks isn’t just partition alignment, but also an issue with RAID-Z’s variable stripe size.
RAID-Z basically works to spread the 128KiB recordsizie upon on its data disks. That would lead to a formula like:
128KiB / (nr_of_drives – parity_drives) = maximum (default) variable stripe size
Let’s do some examples:
3-disk RAID-Z = 128KiB / 2 = 64KiB = good
4-disk RAID-Z = 128KiB / 3 = ~43KiB = BAD!
5-disk RAID-Z = 128KiB / 4 = 32KiB = good
9-disk RAID-Z = 128KiB / 8 = 16KiB = good
4-disk RAID-Z2 = 128KiB / 2 = 64KiB = good
5-disk RAID-Z2 = 128KiB / 3 = ~43KiB = BAD!
6-disk RAID-Z2 = 128KiB / 4 = 32KiB = good
10-disk RAID-Z2 = 128KiB / 8 = 16KiB = good

…but this is mostly irrelevant these days, at least from my understanding. At least unless you are chasing optimal IOPS, which for a file server you probably aren’t.

RAIDz just doesn’t cut it anymore for a very simple reason.

Lets say you have a pool of many drives and one fails. You now have no more redundancy. There is no parity to catch any potential silent corruption that may have occurred, and if another drive dies you lose the whole pool. And this is coming at a time when you are likely putting more stress on your pool than it has seen in a long time, by reading as fast as the system can support, in order to re-silver to the replacement drive.

Guess what. Aging drives + sudden high load = elevated risk of second drive failure, and if that happens all of your data is gone.

The goal these days is that you need to pick if you want to pursue performance/IOPS, or reliability/redundancy.

If you pursue performance, you have very regular backups you can restore from nearby, and you go for mirrors, as many mirrors as you need striped together. This is how you get the best ZFS performance possible for such things as databases, web hosting and other things which require high random access speeds / high IOPS / low latency.

Or you choose reliability / redundancy. This is better for lower intensity file storage. You can make better use of your storage (you don’t have to dedicate at least half of it to redundancy like you do with a mirror) and you gain more drive failure tolerance (if you go above RAIDz) at the cost of a performance hit.

Generally people who know lots about ZFS tend to recommend RAIDz2 VDEV’s in the 6-8 drive size these days, and if that is insufficient, adding a second RAIDz2 VDEV. This way you have some redundancy in place while you are resilvering, and thus you have a lower risk of data loss.

You should still back up your data though. Redundancy is never a replacement for backups.

For some very specific applications (like slow remote backup servers limited in regular operation by WAN speeds) some people will go up to a slightly higher drive count and use RAIDz3, to be able to absorb more simultaneous drive losses, but the downside here is that resilvers - when necessary - will be much slower on a RAIDz3 VDEV than on RAIDz2. Sometimes they can take days.

One thing you can try to do to ease the pain of the initial data transfer, is to create already degraded pools, transfer your data to them, and then swap in the missing drives to create a new vdev in a different size and with different redundancy, but this is highly risky, and really not recommended unless you know what you are doing and understand the risks.

I hope this was helpful.