Fixing my worst TrueNAS Scale mistake!

Hold on I’m a little confused dosent mirror just duplicate the data meaning you could lose theoretically 3 drives assuming they were all on the same vdev but If you lose even 1 drive on both vdevs they’re all gone? I’m fairly new to a lot of this.

Thanks for being honest and introspective about mistakes. We all make them and sometimes we learn the most from them. Hopefully they don’t cost us too much in the process. Great series.

I had 2 servers fail in a space of 2 month because both had 5 disk in RAID5 and during the rebuild a second disk failed. Luckely I had back ups from oen and the other one had a mirror. Since then I do not use anything less than RAID6. Extra HDD is cheaper than the data it holds.

On the previous video you said you spent about 500 Euro for the case. You can find a HP Proliant DL380 Gen9 with 2x Intel Xeon E5-2640v3 8-Core and 32GB ECC RAM that can hold 12x 3.5" (LFF) for around 700-800 Euros used, from a German website and they provide 12 months warranty. A dual port P440ar controller which supports HBA will also cost you around 100 Euro from the same shop.

In any case, nice build!!

yeah i have 10 4tb Drives and i put them in RAID-z2
My PLEX Server running 15 drives is on harware RAID6 but i want to chage that to TrueNAS + HBM some day. and run a 2U for the PLEX Server.

Cheap SSDs and using ZFS?? NONO. Big red flag. You shouldn't use ZFS then.

I am pretty sure that the Lose all your data was the other reason why I picked unraid. I believe that you can still read the data on the unraid drives if multiple drives fail past the parity drive of unraid but I never tested that. Might be a good test for me as well.

Can you help me with a question? I'm setting up a server with truenas scale. I only have three bays available and I was planning to put 3 HDDs of 20tb. What do you think is the best configuration? Thank you so much!

Question, you are using a ssd for storage?

I used consumer SSD's for a while but went back to HDD because ZFS eats up SSD's , way too much write operations cause consumer SSD to degrade in a few months.

if you ask me z2 is quite useless
the fact is chances of failing more than 1 disk at a time is very rare so z1 is still great, you still get warning in SMART to replace the failing disk
when more than 1 hdd failed, what to guarantee that it stays at 2 ??? lmfao!
if a batch of Seagate crap failed then the whole bunch will fail, it doesnt stop at only 2
you must have an offsite backup if you value your data
no amount of parity can save you from a disaster that would kill the entire system, you're just wasting HDDs on that 4 parity

After watching this video..... do I have to be concerned about my configuration. I have 3x8TB and Z1. Which means I can only loose one drive, but I don't want to do Z2 with a fourth drive, as then I only have a efficiency of 50%. So is it really a problem when I have just 3 drives?

Hi Christian

Thank you for the great video. I've been learning a lot from you! It would be great to see a video on automatic backup solutions that are out there. I am currently building a NAS with the main goal of automating backups of my production servers, but not sure where to start or what the best practices are. Thank you so much! Keep well.

Quick Question; is the fault tolerance PER vDev, not over the entire system array as in say a traditional Hardware RAID solution?

as HW it will be one drive ? then i will split it as virtual based ob RAIDZ type ?

Thank you for all your vidoes! How's the health of your SSD's?

I have been using a 2 disk mirror for about 4 years now and I keep a copy on my desktop as well as backing up to a friend's NAS states away from me and the mirror has done well.

You should have used Raid-5. Raid-6 is WAY overkill for 12 drives. Raid is NOT a backup, do not think of it in terms of data loss and in terms of downtime and add an actual backup solution to your environment if you don't already. Cloud solutions make great backup options to get your data off site too, making it safer.

This was helpful for understanding how to optimize larger #'s of discs. You said that you have a 10gbe interface, which one are you using? I ended up with 2 different broadcom cards which would not work and have now ordered a chelsio card. My system started as a core build but my motherboard crashes the installer, so I used scale instead.

Isn't raid 6(z2) better than raid 10 for SSD?

What is this "starch-server"?

Mirrors only. Simple is always the best.

it's not that a second drive fails. it's that if a string of bits fails during parity reconstruction (which is much much more common), some part of your data is gone.

Did they also tell you z2 also has worse performance?

Have you planned to do a new Truenas setup with the changes in ver 23 ?

So what if you have data on the new build? I made this mistake. 8x10TB raidz2 1 vdev 🤦🏻‍♂️ copied all my data from old NAS to this. So now I have to copy it back, reconfigure, then copy back to the new build? Ugh. Any recommendations on best way to copy from new back to old?

There is no 100% fault tolerant config. Even with RZ2 it's entirely possible to lose enough drives to kill your entire data storage. In fact, it happened to me many years ago when a lightning storm hit while I was out of town. The storm compromised my PSU and the PSU killed 5 of the drives and destroyed all of my data.

Since then, I've some to the conclusion that for my personal needs creating pools consisting of 4xHDD@RZ1. I never have less than 2 vdevs in a pool so my IOPS are better than a single disk. The throughput is also good enough for my needs so far. I handle failed disks by having cold spares. Since I'm almost always near my system, if a degraded state were to ever show up, I can just power down and swap the drive in a matter of minutes. It would be different if the server was in a datacenter somewhere that isn't instantly accessible but for most home users, that's simply not the case. I also have a duplicate identical system at my son's house. The 2 systems are synced so the data is theoretically always backed up.

All this is to say, I personally believe that sacrificing 2 disks per vdev in a home environment is a waste of storage space and money. As long as you have 1 or 2 cold spares and a good UPS/protection circuit, you should almost never be put in a situation where the problem can't be resolve immediately.

AND? some new film coming? with conclusion after 1 year?

My setup, nas1, 4x18tb raidz1, nvme meta and cache, 8tb mirrors, 1tb apps and vms. Nas 2 8x8tb raidz1, backup. 10 gbe between both. 27 tb backup was 15 hours on 1st sync,

If you haven't done so yet, go into "Data Protection" and schedule SMART tests, short and long. Maybe do the short say bi-weekly, and the long monthly or so. You can also setup so system errors can get emailed to you. So then if you have a SMART failure and it gets logged it will also get emailed to you.

good thing i find thix video just before i get the old used pc i mgetting for my nas , so i will be sure not to do the mistake 🙂
im getting an old i5 7th gen with 16 gig ram(ill upgrade it over time to 64(max i can have on that mother board)
and i bought 3 ssd 1 t and a 250 gig ssd nvme for the cache , i have already in it a 112 gig ssd that ill set for my boot installation .
i will use it a s cloud (next cloud,) my plex server, and maybe other things that ill find over time i already got a domaine name to access my cloud ect,
and alreday have a cloudflare set up too

and i ll save money for later get a bigger sever that ill install proxmox and install vm on it and ill see if i keep that one as a nas server or use only for some thing else
adn mistake happen to any one

While having a dedicated VM pool on SSDs is absolutely nice, as an intermediate step, you could have added a L2ARC and SLOG device to your main pool and run the VMs from there. For syncronous IOPS, the SLOG devices is a huge deal (eg for databases) and the L2ARC (especially when marked persistent) will make your VMs feel snappier when reading from their respective disks. In principle, adding more RAM also helps by a lot, but the persistent L2ARC is especially helpful, when you reboot TrueNAS and then want to spin up your VMs: ARC in RAM is then not yet populated but L2ARC was, giving you SSD IOPS instead of HDD.

Fault tolerance is only dictated by how much downtime you can afford, not preventing data loss. If you have everything backed up and have the time to rebuild and recover then there's nothing wrong with raidz1.

Yes, this is a mistake imho.
People that don't have at least an off-site location AND a decent secured cloud backup solution don't even have to start thinking about raid.
What those pleb comments maybe never told you is that raid isn't for data security OR integrity at all. It's just for availability.
A fire burns your house down? Good luck with a shitty raid.
Water floods your house? Good luck with a shitty raid.
A thief steals your NAS? Good luck with a shitty raid.
A cryptolocker encrypts your whole environment? Good luck with a shitty raid.

Raid is useless for most people, yet most people hype it. It's absolutely mind boggling how many people "do IT" yet don't understand the basics of it.

It's laughable to discuss the consequences of a 2nd drive failing during rebuild while having ZERO protection against a real issue.

If videos aren't prefaced with:
yes, you do have an off-site redundant system running and
yes you do have scheduled secured cloud backups and
yes you do have rollback capability then it's nothing but a fail and has nothing to do with data security.

The best way is don't use ZFS for data backup, use unraid instead it is supper safe and very fast

Any update on how those SSDs were?

man im glad i found your video. this is almost the EXACT setup i have running. with 12x 4tb drives. i dont mind sacrificing 33% because the security is deff worth it, but full mirroring just felt overkill. in fact, despite me skipping directly to the part i needed, im going to go ahead and rewatch the full video to make sure i didnt miss anything. maybe you had a mistake i can learn from

Mh, another nonsense video that is talking about theoretical numbers only without ever checking anything and giving actual real-world numbers.

Christian, as a fellow German TrueNAS user I gotta tell you: Raid is not a backup, and having said that, wasting a total of 4 out of 12 disks for redundancy by losing almost a third of bandwidth and only gaining twice the amount of IOPS in my personal opinion is just not what I'd be aiming for.
If you happen to not use the storage for things like VMs, databases and APPs, you don't need those IOPS, you need throughput for your files, which you gain by having more spindles with actual data. In that case, a RaidZ3 would have given you more bandwidth, actually MORE security from data loss (although I personally think if you have quality drives and not some shucked stuff, Losing more than 2 disks out of 12 at the same time is EXTREMELY unrealistic. Just do your scrubs as a good boy does, and a Z1 will be more than adequate for a 12 disk array.
If you happen to use that space for things where IOPS do count, I would be going for a stripe of 3 Z1s with 4 disks each, or if you want more, then go for a stripe of 4 Z1s with 3 disks each. That is plenty of redundancy considering you will perform a backup AT LEAST every week anyway, you will lose less or the same in volume and both the bandwidth and IOPS will improve (IOPS significantly).

Thanks for the very clear explanations. I'd love to see your take on backup and recovery of ZFS pools. That would make a good video. I suspect the backup topic would elicit plenty of critical feedback as well. :)

Maybe just maybe you're using the spinning drives vs ssd or nvme, hence, failure is imminent.

I'm configuring my home Truenas setup. Started with raidz1, and after some reading, I went to raidz2. I have 5 drives in a single vdev.

Please replace your Sata SSDs with Samsung EVO or Crucial MX. I bet yours will become slower than old spinning disks as soon as you fill it. That as my case anyway, and I am much happier with good SSDs from reputable brands. I may be too late watching your video, but if you have any news on that please let me know =). And thanks for the info!

I love how he says a second drive dies "sometimes". lol. It has happened to me a few times over the years with the WD Reds, which is why I always use at least raidz2.

I don't understand why people absolutely need one big chunk of data space on theirs filservers, why put all disc's in a single pool?
Why don't have your two raidz2 separately? If a worst case disaster occurs you are only loosing 50% of your data anyway.
All it take is some planning of your usage and splitting up your data.
But understandable if you not are dealing with wery big files and stuff.

I am building a sever with FOUR 4TB HDDs (this would be my archive pool that I need max redundancy for, so I would only have 4TB max storage), and a second pool of FOUR 500TB SSDs that I would use for max speed and performance for temporary uses. (Combining them for almost 2TB temporary work space?) What would you recommend settings for this situation?

12 drives in only one single raidz1 vdev is nuts! you are multiplying by 4 the chances of a catastrophic data loss.

Try using SAS drives (instead of SATA), which could perform R/W I/Os simultaneously. Even though it is not technically possible to read and write at the same time when each disk has only one set of heads per drive, controller does not have to wait for the data to be returned to the controller in order to proceed further.

I put 8 HDDs in single vDEV RaidZ2 and have only 250MiB/s write transfers...

christian, i see ur using 12 drives @ 4tb.

just wondering wouldnt it be cheaper electricity wise running FEWER hard drives?

personally i am keeping it 4 x 3.5''' hdds max (im using 12tb drives). And 3 ssds tops (2 as a mirror for vm/docker containers, and a single drive for truenas os).