About two months ago, I was putting the finishing touches on assembling and burning in the DIY NAS: 2022 Edition. Since then, I’ve been working on a couple tasks:
- Build some confidence in the hardware that I purchased.
- Establish some confidence in the two latest TrueNAS SCALE release candidates.
From the beginning, my plan was to create a fresh install of TrueNAS SCALE, export my 7-drive pool from my old NAS, import it into the DIY NAS: 2022 Edition, and rebuild everything from scratch. In addition to that, I also wanted to move my virtual machines from my Homelab server onto this new DIY NAS.
Since building it, the DIY NAS: 2022 Edition has been stable. However, I’ve had one hiccup. I experienced a single read error on two different hard drives one night (2 read errors total). One of those errors pushed the total error count of one of the two hard disk drives past a threshold, and ZFS kicked the drive out of the pool.
Cue ominous music... pic.twitter.com/vcFMWpAs1I
— Brian Moses (@briancmoses) January 11, 2022
Out of an abundance of caution, I ran long SMART tests on both drives, examined the SMART data, and felt confident that I would be able to put the supposed degraded drive back into the array. After the resilvering was complete, I ran another set of long SMART tests on both drives, and haven’t had any issues since.
In the couple weeks since they occurred, I haven’t seen anything like it since, and I’ve been using my NAS pretty heavily since then. I’m not at all concerned about these errors. I often use my blog as a personal reference, so I’m noting them here for when—or if—they occur again.
TrueNAS SCALE does everything that I have been using TrueNAS CORE for
Above everything else, I needed TrueNAS SCALE to be able to replicate all of the features of TrueNAS CORE that I rely on. In the event that I couldn’t get SCALE to do this, I was not going to make any compromises and I planned to revert back to CORE immediately.
I’ve been using my DIY NAS as my primary storage for years now. I had a suite of snapshot tasks for the data stored on my NAS, SMB shares, an NFS share, cloud sync tasks to back up my NAS to Backblaze, a VM operating as a Tailscale Relay Node, and Nextcloud running in a VM and accessible via Tailscale.
It took me a handful of hours spread over a few days to recreate all of this from scratch on the DIY NAS: 2022 Edition. Setting everything from scratch was effortless, thanks to the SCALE UI. The folks at iXsystems have done a commendable job at making the management interface consistent between SCALE and CORE that rebuilding my NAS from scratch was a straightforward task.
TrueNAS SCALE quickly demonstrated to me that it was up to the task to meet all of the needs that FreeNAS and TrueNAS CORE have done so well since building my very first DIY NAS.
I was not able to consolidate all of my Virtual Machines onto TrueNAS SCALE
Between my Homelab and TrueNAS CORE machines, I had a few Virtual Machines: a Tailscale VM that I used as a Relay Node, Plex, and my Home Assistant virtual machine. I chose to deprecate the Tailscale Relay Node VM. I installed the “official” Plex app, I had some misadventures with the Nextcloud “apps” (more on that later), and ultimately wound up creating a new Nextcloud Virtual Machine to run on the DIY NAS: 2022 Edition.
After all of that, the only thing left running on my Homelab machine was my Home Assistant VM. I have a lot of Zwave home-automation devices which are controlled by a Zooz S2 USB stick (ZST10 700). On my Homelab machine, I’ve been passing through this single USB device to the Home Assistant virtual machine. But when it came time to do the same under SCALE, I learned it was going to be a bit trickier.
Currently, you cannot pass through an individual USB device using the SCALE interface. You can do it from the command-line, but anything you do at the command-line will get wiped out the next time the virtual machine restarts. The supported workaround is you need to pass through the entire USB controller to that virtual machine. Technically, I could potentially make this work but I’m currently unwilling to implement it. Redirecting the entire USB controller doesn’t seem like a very good solution. I’m fairly certain that other USB devices (keyboard, mouse, my UPS, etc.) that I wanted to plug into my NAS would get redirected to a virtual machine.
For the time being, I’m giving up my hope of decommissioning my Homelab server. I will continue to use it to host virtual machines that TrueNAS SCALE can’t support until SCALE resolves these requests. From the speculation I’ve read, it’ll be quite a few months before SCALE will support passthrough of an individual USB device to a virtual machine.
- NAS-108800: Assigning Host USB device to a Guest VM in SCALE
- NAS-108713: SCALE: Mount USB Devices to VM
TrueNAS SCALE’s Apps Experience
I was pretty excited that TrueNAS SCALE was leveraging containerzation–even more so that they were going to try and make it simpler via its apps. I was pleased when Nextcloud and iXsystems announced that Nextcloud would be officially supported on TrueNAS. I think the concept of companies collaborating together like this will have a lot of benefit to their users.
I installed two official apps from the TrueNAS SCALE catalog; Plex and Nextcloud. As I mentioned previously, Plex was up and running without any issues in mere moments. The official Nextcloud app was a bit different.
Because my apps and VMs are being installed to an SSD mirror, I needed to use the Nextcloud option to use an external data source (a dataset on my 7x drive array). But anytime I picked the options to use an external storage location, the file-permissions inside the Nextcloud folder wound up being incorrect. It took a little bit of tinkering inside the container to change the file permissions for both the Nextcloud directories as well as my external data in order to get it working.
Because I worried that any future updates of the Nextcloud app would mean I’d need to be updating those permissions, out of curiosity, I thought that I’d try the Nextcloud app from a different catalog: TrueCharts.
TrueCharts
TrueCharts is a separate organization that has created an impressive catalog for use inside TrueNAS SCALE. Initially, I was excited because it appeared that there was a pretty substantial amount of documentation, including some guided walkthrough videos on YouTube. But ultimately, I found the documentation and videos to be pretty spartan, and I was frustrated by the number of bad links that all 404’ed on TrueCharts’s website.
Regardless of the state of the documentation, I was able to get their Nextcloud app functional on my NAS using my external storage. However, what I really wanted to do was share my Nextcloud app with friends and family using Tailscale, and that wasn’t working for me. I never thought this was TrueCharts’s fault, but I lacked the expertise to get to the bottom of it on my own, and I was going to need some help.
I’ve been a member of the TrueNAS Discord server for a while before installing SCALE. I was curious about listening in on what people had to share about SCALE. I was especially curious to hear about what people were doing with “apps” under SCALE. One of the recurring themes was users who were frustrated with how TrueCharts’s Discord server was being moderated. I mostly chalked it up to people being frustrated, but the repeating pattern was troubling.
When I ran into issues with TrueCharts’s Nextcloud app not listening on my NAS’s Tailscale IP address, I experienced this firsthand. Aware of the emphasis on following rules, I read through everything that I could, scoured each channel’s pinned posts, and searched all the past threads before creating my own support thread. Within a few moments, I was told what I was doing was hacky, unsupported, my support thread was archived, and I was told to go ask my question in a different “experts” channel. I quickly found that I lacked the ability to even post in the experts channel, so I messaged the moderator who gave me those directions. Their response was to scold me for using direct messages. Without any other options, I decided to delete the TrueCharts Nextcloud app and leave their Discord server.
A week or so later, I got a chance to privately discuss this experience with a different moderator of the TrueCharts Discord. It was a pleasant conversation, and I appreciate their admission that it was handled poorly and must have been either a bug or a mistake.
My experience with TrueCharts sapped any interest that I had in their app catalog. Even worse, it had a negative impact on my opinion of TrueNAS SCALE. I hope that the TrueCharts community matures, grows, and evolves to the point where experiences like mine are rare. However, that journey is just beginning for TrueCharts. Until they complete that journey, I’d recommend exploring all other possible options before relying on anything from the TrueCharts app catalog.
Things seem to break in TrueNAS SCALE when running Tailscale on the host
One of the biggest reasons that I’ve interested in TrueNAS SCALE is to install Tailscale on the host and use Tailscale to access my NAS from outside of my network and to share the VMs and apps that I’m running on my NAS. In migrating over to my new NAS, I learned that I wasn’t going to be able to share my Tailscale relay node with my friends and family. To accomplish what I wanted to do, I’d need to install Tailscale on the host itself.
It’s unsupported, but it is possible to install Tailscale on the TrueNAS SCALE host, which is an improvement over CORE. For a while, it even seemed to be working okay. I could access the official Plex app and all of SCALE’s hosted services (the web-management interface, SSH, SMB shares, etc.…) but in the days after installing Tailscale, I wound up running into several other problems:
- Tailscale got wiped out when I upgraded TrueNAS SCALE from 22.02-RC1 to 22.02-RC2
- The Nextcloud app from TrueCharts only listens on the IP address of the host itself; any traffic to the host’s Tailscale IP is ignored.
- I had one instance where my SMB shares stopped being accessible until I stopped Tailscale.
- Others reported issues with Tailscale preventing Kubernetes from starting up
As I've been tinkering with my new #DIY #NAS, it's been interesting to realize how quickly implementing @Tailscale has become essential--and how willing I am to move away from thing(s) that make Tailscale's use more difficult. https://t.co/CGgCVp8Gz3
— Brian Moses (@briancmoses) January 6, 2022
This has been a tremendous disappointment to me. Tailscale has been so easy that I figured if I could get it installed on the host, everything on the host would be accessible on my Tailnet. But that didn’t turn out to be the case. I knew what I was trying wasn’t supported and I wouldn’t get much help for it, so I searched for a Tailscale feature request(s) on iXsystem’s JIRA page to upvote and found this:
If I want to host something on my NAS and share it via Tailscale, it will need to go in its very own virtual machine. This is how I was handling sharing things with TrueNAS CORE, and I’m content to keep doing it while using TrueNAS SCALE. I’m not thrilled with this, but I can accept it—for now.
Will Brian be using TrueNAS SCALE in 2023?
For as long as I have been using TrueNAS CORE (or FreeNAS), it has met all of my requirements. I’ve never even seriously considered looking at anything else. I like the ZFS file system, I like how easy TrueNAS CORE makes it to manage ZFS, and I remain excited about the possibility of completely consolidating my Homelab and NAS machines onto one platform.
But the minute I saw the “hyperconverged” buzzword in SCALE’s marketing material, my expectations for SCALE shifted a bit. After installing SCALE, I expected that I’d be able to retire my Homelab machine. I also expected that I’d be able to access and share all the applications hosted on my TrueNAS machine using Tailscale. Unfortunately for me, those expectations haven’t been met—at least, not yet. If the following happens this year, I’ll be really excited:
- Incorporate support for Tailscale on the host for accessing/sharing the core SCALE services and any apps.
- Enable features to leverage KVM’s pre-existing support for passthrough of unique USB devices to a Virtual Machine.
- Expansion of the list of “official” apps in the TrueNAS SCALE catalog.
- A maturation of the TrueCharts community to err on the side of inclusion, rather than exclusion.
- Development of other catalogs of apps to compliment and compete with the current catalogs.
At the end of this year, I fully expect that I’ll still be using TrueNAS SCALE. By the time 2023 rolls around, I fully expect many of these items either have been resolved or a roadmap laid out for their resolution under SCALE.
Final Thoughts
Don’t let the words I’ve invested in the two areas that I’ve been disappointed with TrueNAS SCALE mislead you. Overall, I’m incredibly satisfied with TrueNAS SCALE. It fell short on a couple of my expectations, but I was also expecting that SCALE would far exceed what I have historically asked my own NAS to do. If I were asked to make a recommendation, I’d encourage others to take a look at TrueNAS SCALE first. I think its hardware support and its unrealized potential make it a great choice for DIY NAS builders today.
That being said, SCALE did not live up to my expectations, too. If my expectations aren’t being met by the end of this year, it will be time to seriously consider alternatives like UNRAID, Proxmox, or even building my own homebrew server from scratch.
After reading about my experience with TrueNAS SCALE, what do you all think? If you’re a TrueNAS CORE (or FreeNAS) user, are you excited about switching over to TrueNAS SCALE, or are you sticking with TrueNAS CORE? If you’re a prospective new builder, do you have a preference for SCALE or CORE? I’m interested to hear what you think down in the comments!