CENTAURI
A self-hosted infrastructure lab: 38 containers on ~109 TiB of dual-parity ZFS, in a drive enclosure I fabricated myself.
A household that wanted its photos, files and media to stay available without renting them from four different subscriptions — and, honestly, a desire to run real infrastructure rather than read about it. It serves people who will tell me directly when it is down, which is a more demanding form of uptime pressure than it sounds.
What it runs on, and why
- TrueNAS SCALE
- ZFS as a first-class citizen with a management layer I do not have to write.
- ZFS (RAID-Z2)
- Dual parity, checksummed integrity, and snapshots that cost almost nothing.
- Docker
- 38 containers across TrueNAS-managed apps and hand-deployed stacks.
- LSI SAS3008 HBA (IT mode)
- Presents disks directly to ZFS. Hardware RAID here would actively hurt.
- Tailscale
- WireGuard mesh for the admin plane, with no ports open to reach it.
- Cloudflare Tunnel
- Publishes a small set of services outbound-only, surviving any NAT situation.
- nginx-proxy-manager
- Reverse proxy and TLS termination behind the tunnels.
How it fits together
Read this diagram as text
- Internet → ISP router (WAN) — crosses to the public internet
- ISP router → Switch
- Switch → Centauri (single link)
- Centauri → Docker
- Centauri → SAS3008 HBA (PCIe)
- Docker → Cloudflare Tunnel (outbound only) — crosses to the public internet
- Docker → Tailscale (WireGuard)
- SAS3008 HBA → tank (SAS)
- 01
A 3 Gbps symmetric line terminates on an ISP-supplied router whose firewall rules I do not own. There are no inbound port forwards on it — not one — which is the single most important fact about this topology.
- 02
From there a single 2.5 GbE link runs through an unmanaged switch to the host. There are no VLANs and no segmentation at layer 2. This is the honest weak point of the design, and it is also the bottleneck: the array reads faster than the wire.
- 03
The host is a Lenovo ThinkCentre. The disks are not in it — an LSI SAS3008 HBA in IT mode reaches out over SAS to a JBOD enclosure I built, so ZFS addresses all eight drives directly with nothing pretending to be a RAID controller in between.
- 04
Everything above the storage runs in Docker: 38 containers split between TrueNAS-managed apps and hand-deployed stacks.
- 05
Access divides into two planes. Administration — the TrueNAS UI, SSH, Portainer, logs — is reachable only across a Tailscale WireGuard mesh. A deliberately short list of services is published outbound through Cloudflare Tunnel for family who are not going to install a VPN client.
Decisions, trade-offs, failures and measurements. Everything below is collapsed by default because most readers do not need it — and expanded, because if you are still here you probably do.
DECISION_LOGWhy, and what it cost
Why, and what it cost
Why rebuild the pool instead of replacing drives in place?
The pool started as two RAID-Z1 vdevs, one of them 20 TB drives — a layout I had already written up publicly as my worst decision, because a single failure at that capacity means a multi-day resilver hammering the survivors, and a second failure in that window takes everything. ZFS cannot convert a RAID-Z1 vdev to Z2, cannot merge vdevs, and cannot remove a raidz vdev from a pool. In-place replacement would have bought capacity while preserving the exact topology I was trying to escape. Destroying the pool was not the aggressive option; it was the only route to the target.
Everything had to come off the array first. Photos and configuration were backed up; the media library was not, because it is the one category I can rebuild. That classification was written down as a principle months earlier — this is the week it cost something real.
Why an HBA in IT mode and a JBOD, rather than a NAS appliance?
ZFS needs to see raw devices. A RAID controller sitting between ZFS and the disks hides exactly the information ZFS relies on to detect and repair corruption, so IT-mode firmware is not a preference here, it is a requirement. Building the enclosure rather than buying a shelf used metalworking skills and shop access I had from my job at the time, and kept the drive count open-ended instead of capped by an appliance.
No vendor support, no hot-swap backplane, and the enclosure runs on its own PSU with no power sequencing — so the drives must be powered up before the host or the HBA enumerates an empty bus. Nothing here recovers from a power event unattended.
Why no SLOG or L2ARC?
A SLOG only accelerates synchronous writes. Almost everything this system does is SMB and async writes from containers, which would never touch it. An L2ARC keeps its index in ARC, so adding one to a memory-constrained host can make things measurably worse. Buying either to look thorough produces an SSD that does nothing.
The one workload that would genuinely benefit is the 1 TB zvol exported over iSCSI, which is exactly the sync-write case a SLOG exists for. That is the honest counter-argument, and I have accepted it rather than answered it.
Why remove the GPU rather than keep it working?
TrueNAS SCALE 25.10 switched to NVIDIA open GPU kernel modules, which depend on the GSP — a coprocessor introduced in Turing. Pascal does not have one, so the card became unsupported. This was an architectural dependency, not a card that got old. Community rebuilds of the legacy proprietary driver exist and would have kept it alive.
Maintaining out-of-tree kernel modules against every future TrueNAS update, for one 4 GB card, was more ongoing work than the card was worth. Immich machine-learning passes now run on CPU and are slower for it, and the local LLM containers were shut down rather than migrated — they were never more than a toy at 4 GB of VRAM.
INCIDENT_LOGWhat has actually gone wrong
What has actually gone wrong
Symptom
The pool went DEGRADED and stayed unstable for days. Individual drives cycled between DEGRADED, FAULTED, REMOVED and OFFLINE, and the fault kept moving between devices rather than staying on one.
Impact
Intermittently usable rather than cleanly down. The pool was sometimes ONLINE while resilvering, but also hit suspended I/O and full storage freezes. Pool import and export operations became extremely slow or hung entirely. Every service backed by the array was affected for the duration.
Investigation
- 01Started from the TrueNAS email alerts and `zpool status`, which showed the pool degraded and resilvering with disks changing state.
- 02First working theory was a single failing 20 TB drive — the obvious reading, and the wrong one.
- 03Physically pulled the suspect disk. This made things worse: TrueNAS immediately reported "ZFS pool I/O is suspended", which temporarily made the problem look more like a pool or filesystem failure than a hardware one.
- 04spent time questioning whether SATA drives were appropriate at all, and seriously considered replacing all eight with SAS units. Also chased breakout cables, power splitters, insufficient PSU capacity, HBA cooling, firmware, and PCIe instability.wrong turn
- 05Moved from the ZFS layer down to the kernel. The `mpt3sas` logs showed multiple devices being reset and removed through the same controller path — including disks on HBA PHYs 4, 5 and 7.
- 06Correlated the failures rather than counting them: at least four distinct drives reported problems over several days, and affected disks could return online and resume resilvering. That is not how several independent mechanical failures behave.
- 07Noted `task txg_sync blocked for more than 120 seconds` in the logs, which explained the system freezes — ZFS was waiting indefinitely on I/O beneath it.
- 08Reassessed the timing. The problem appeared after the 20 TB upgrade, which made the new drives look causal. More likely, sustained resilver load simply exposed a controller path that was already marginal.
Representative kernel output
mpt3sas_cm0: log_info(0x31110e03) sd 0:0:4:0: device_block, handle(0x000c) sd 0:0:5:0: device_block, handle(0x000d) INFO: task txg_sync:1042 blocked for more than 120 seconds.
Root cause
The HBA had failed, not the disks. Simultaneous errors across multiple drives on the same controller point at the shared component — the card and its breakout cables — rather than at the drives themselves.
Fix
Replaced the unstable HBA with a known-good LSI/Broadcom controller running IT-mode firmware. The pool resilvered and returned to a healthy state on its own.
Prevention
Honestly: limited. There was no monitoring in place that would have caught a controller degrading before it started taking drives offline, and there still is not. What did work was dual parity — RAID-Z2 absorbed the instability long enough to diagnose the real cause instead of forcing a panicked rebuild.
Lesson
When several drives fail at once, suspect the thing they have in common. The evidence that actually settled it was correlation — failures moving across devices on one controller path — not a clean bill of health on any individual disk. Worth noting the limit of that reasoning too: no clean SMART output from the affected drives survives, so this was never "SMART proved the disks were fine". It did not need to be. SMART frequently cannot see HBA, cable or link instability at all, which is exactly why the correlated-failure pattern is the stronger signal.
TRUST_BOUNDARIESWhat is defended, and what is not
What is defended, and what is not
Administration
TrueNAS UI, SSH, Portainer and logs are reachable only over a Tailscale WireGuard mesh. Nothing on this plane is published.
Published services
Jellyfin, its request front-end and Wizarr are exposed via Cloudflare Tunnel — an outbound connection, so no inbound rules exist on the router.
GapThey are protected by their own application logins and nothing else. There is no identity layer such as Cloudflare Access in front of them, largely because it adds friction for non-technical users. That is a reason, not a defence.
Network segmentation
None.
GapThe switch is unmanaged, so there are no VLANs and no layer-2 isolation. Every device shares one flat network. Segmentation would require replacing the switch, which is on the same list as the 10 GbE upgrade.
Torrent traffic egress
The download client runs in an image with an integrated VPN and a strict kill switch — if the tunnel drops it has no network at all rather than falling back to the WAN. It is the only container here with an egress path of its own.
Backups at rest
The TrueNAS configuration bundle contains secrets and keys, so it is encrypted before it reaches third-party cloud storage.
Power and physical
None worth the name.
GapNo UPS. Eight spinning disks and an active pool are exposed to abrupt loss, and with no SLOG the sync writes that do occur land in the in-pool ZIL. Combined with manual power sequencing, this is the least mature layer of the whole build and the next thing I would spend money on.
MEASUREMENTSNumbers, and where they came from
Numbers, and where they came from
`zpool status` after a scheduled scrub of 111 TiB allocated, 2026-09-02.
`zpool list -v`.
`zpool status`.
Theoretical, not measured — the practical ceiling of a single 2.5 GbE link. An 8-wide RAID-Z2 of CMR drives exceeds this sequentially, which is why the NIC is the constraint on every transfer.
Weekly TrueNAS Cloud Sync of photos and configuration. Anything added since the last successful run exists only on the array. Media is deliberately outside this scope.
POST_MORTEMWhat I would do differently
What I would do differently
- ▹
Do not opt into the pre-release update channel on the machine the household depends on. I selected Early Adopter and found out it is a one-way door — once you are running a build newer than stable, the General channel is not selectable any more, and going back means a clean install and a config restore.
- ▹
Buy the UPS before the tenth drive. Power is the least mature part of this build and the only one where a single event could cost me data rather than time.
- ▹
Automate the power sequencing, or stop needing it. Requiring the JBOD to be powered before the host means nothing recovers unattended, which quietly turns every outage into a physical-presence problem.
- ▹
Application state deserves the same care as the data. `app-pool` is a single non-redundant SSD at 80% capacity holding every container configuration, sitting next to a data pool with dual parity. The asymmetry is not defensible; it is just what happened.
- ▹
Watch the 80% line. The pool is at 76% and ZFS gets meaningfully worse at allocation past 80%, so the decision about pruning or widening is already overdue rather than upcoming.