Computer Related > Whoops, there goes the NAS Computing Issues
Thread Author: tyrednemotional Replies: 18

 Whoops, there goes the NAS - tyrednemotional
I had an alert from my NAS at lunchtime. One of the drives in a mirrored RAID array had tripped a "bad sector" trigger on the SMART monitoring.

It is continuing to work, dependent on the non-failing disk mirrored data.

Given my monthly hardware scan of the disk had just started, I don't think the alert was a coincidence.

In theory, I can just power off, remove the failing drive and replace it with one of the same or greater capacity, power back on and it will repopulate the new disk.

In practice, it's been running 16 hours a day for 6½ years, and I can't easily source a like-for-like replacement (as in all mirroring, the recommendation, though not an absolute requirement, is to use "identical" drives), I'll be replacing both drives (protecting agains similar failure on the second).

It's a bit more complicated to do that, made more so by the fact that I can buy bigger capacity drives (on offer) than the same-sized units. The process of resizing is, however, supposedly "built in" to the NAS OS, so we'll see.

Dependent on the arrival time of the replacement drives, that looks like much of tomorrow and/or Friday gone! (rebuilding a 4TB mirror effectively twice takes a good amount of elapsed time).
 Whoops, there goes the NAS - Kevin
Shirley just a case of swapping the disks one after the other and letting it do it's thing in the background?

Or does it give you a progress bar to watch ;-)
 Whoops, there goes the NAS - tyrednemotional
If you swap a larger disk in, it will mirror to the smaller one's capacity.

I'm not sure that then swapping in a second larger one will expand the array to the new, maximum size or not.

If I'd bought same-sized disks as current, then I'd simply do a two-stage swap, but since the NAS has a specific process for expanding the array (which will involve a two-stage swap in the script) then I'll do it their way ;-)
 Whoops, there goes the NAS - Kevin
How much data is written in a day or so? Will it let you make RAID read-only, break mirror and copy data to bigger disk, insert 2nd disk and remirror?

Still two stages but more control over what it's doing if you can afford the read-only time.
 Whoops, there goes the NAS - tyrednemotional
>>Will it let you make RAID read-only, break mirror and copy data to bigger disk, insert 2nd disk and remirror?

...the inbuilt process essentially does that copying but keeps the NAS available. (It would be easier/quicker if the disks were hot-swappable, though).

>>Cant you upgrade your NAS in total and recover from your tape silo? replicate the data to an upgraded NAS via
>> your SAN?

...let's keep it simple, eh! Ultimately, I can recover all the user data from a 3TB usb portable drive (which was relatively up-to-date before the incident, but is now current). I'd prefer not to have to re-initialise/rebuild the NAS system and customisation, though, so the prescribed expansion method will do.

(I've actually bought a good number of tape silos in the past, StorageTek ISTR. You'd get almost as many of my NAS in one as you would cartridges)
 Whoops, there goes the NAS - Kevin
Boils down to whether you trust their resizing utility then.

Feeling lucky?
 Whoops, there goes the NAS - tyrednemotional
...nothing much to lose, except time. Multiple ways of recovering if it goes belly up.
 Whoops, there goes the NAS - Zero
Cant you upgrade your NAS in total and recover from your tape silo? replicate the data to an upgraded NAS via your SAN?
 Whoops, there goes the NAS - Kevin
Not until he's upgraded the firmware on all his FC adapters and the SAN Switches.
 Whoops, there goes the NAS - smokie
I accidentally formatted my 8Tb USB drive instead of a new 16Gb SD card for the Pi one day a year or so back. Really careless mistake that was over and done with, and irreversible, in a split second.

Luckily the disk was mostly only holding backups of other stuff, or stuff that was backed up elsewhere, though it was fairly full (currently about 840Gb free). Took me days and days to recover IIRC (much of it unattended though).
 Whoops, there goes the NAS - tyrednemotional NAS holds data that is largely available elsewhere, so could be rebuilt, albeit taking time and having to remember all the settings.

I've taken an extra backup of the contents, however (can't have too many backups!).

Other than an automated repository for backing up data from the various user machines around the house, it's main task is to stream my music to various devices including Hi-Fis, and to play photos and videos on demand from the TVs.

I wouldn't be without it. The "fire and forget" backup regime it enables ensures that, in the event of a disaster, all user data can be restored to a PC up to the state as at the last early evening backup (taken daily).

It had been used to restore but a single file that had been deleted in error, up to the point where the HDD on SWMBOs laptop borked terminally with no warning. It was worth the expense and effort subsequently to be able to restore all the data to a point about an hour before the event. (there was an awful lot of "research" data on there).
 Whoops, there goes the NAS - smokie
"can't have too many backups!" - this, absolutely.

User PCs (just SWMBO now) are in the destiny of the user - they have the mechanism to easily back up their data as often as they like to the NAS. Once a week the NAS is backed up to my USB drive and once a month to another USB drive - then once when I remember I ask my daughter to bring home the offsite which holds the most key 3Gb or so (the space used is mainly photos but the important stuff is small in comparison). Oh, and there is the old NAS which is mostly offline but has backups of backups, plus some bulky stuff which it would be pointless clogging up a disk with - e.g. a load of tutorial videos downloaded ages ago for a hobby I've not really yet begun.

Some of my own stuff which is updated daily is backed up automatically each time my PC is turned on, other than that I do my end of the backups manually. It's a bit like me hand washing the car - gives me the chance check over the general state of things and keep tabs on how things are. I spend a fair bit of time at my computer so it isn't a chore.

The slow thing is imaging the SD cards in the Pis. I do that the day before the main backup day so they are all included, though one is fairly static and doesn't get done that often. I am trying to transition the Home Assistant to an SSD which will prevent me imaging that, as it's all or nothing with the SD card image software. I know there are probably different ways to do it but I finally managed to successfully restore Home Assistant backups in testing the other day, which I'd never managed before, so that won't need doing once I can prove to myself that it is robust and easily recoverable. (The HA restore software is nasty, it doesn't even let you know it's started, let alone report progress, or completion. Plus I removed a particular troublesome integration the other day which seems to have improved performance as well as reducing snapshot size by multiples.)

I'm sure some will think it sad but we all have hobbies, and our home IT setup is one of mine :-)
 Whoops, there goes the NAS - tyrednemotional
...well, I've started the replacement/expansion using larger disks.

The inbuilt process is designed for "hot-swappable" disks. My NAS isn't physically designed for that, though others by the same manufacturer are. Accordingly, the process is being run with the NAS disassembled. (I have hits on forums that people have undertaken the task in that way, and the swap worked when prompted, and now the new larger disk is resynchronising (rebuilding the mirror) with the remaining original).

From past building experience, it was expected to be slow, and it is - painfully so! At the end of this phase it's swap the other disk, (for a larger one)and resynchronise in reverse.

It then follows to expand the volume by using and synchronising the newly available space (which again will take time).

It can't be turned off in the interim, so it looks like I might be pulling an all-nighter. (brings back distant work memories of major recoveries of business-critical databases. I so miss that, not! ;-) ).


it is not impossible that, at the end of that process, it won't restart properly, but come up in an "initialising" state. There appears to be a quick and easy trick to correct that - we shall see!
 Whoops, there goes the NAS - tyrednemotional
...well, 17 hours later, swapping out the two disks and replacing with a pair of larger ones, it's all back as it was.

Mirror rebuilt and data retained, then the image expanded to fit the bigger disks. A few "iffy" moments but generally ok.

i) Having rebuilt, but still using the original size, the process invites you to expand the RAID array into the extra space by selecting "Expand", but the button to do so wasn't on the taskbar!. After a few choice words, and about 15 seconds, the taskbar magically refreshed itself and it was there!

ii) It did come up at the end in an initialising state (which, if continued would destroy all the rebuilding and the data. The quick and easy (but far from intuitive) trick noted above resolved that - instead of letting it go looking on the web for updates to its firmware itself, which starts the full reset process, you manually update the current firmware with a (the same version) locally held copy. It reboots after installation and, hey presto, it is now absolutely fine, realising it doesn't need to initialise. If I hadn't found a tip-off on this, things would have been very frustrating.

Because of the need to hot-swap and keep powered-up, the unit was out of its case for the duration - a bit of a nuisance, but it worked.

....and throughout, as it should be, the device was fully functional on the network, albeit most of the time from a singe-disc, not a mirrored pair. (I minimised activity anyway, but in the midst of it it updated one of my streaming applications (twice, different versions! - bug in the first update I think.))

 Whoops, there goes the NAS - smokie
Sounds like a good job well done!
 Whoops, there goes the NAS - Zero
what did we do before we had TB of data to worry about........
 Whoops, there goes the NAS - tyrednemotional
...I counted each byte out......and I counted them all back in again...

 Whoops, there goes the NAS - Kevin
I hope you didn't install 8TB Hitachi.
 Whoops, there goes the NAS - tyrednemotional
...WD 3TB came out, Seagate 4TB went in.

I look at backblaze from time-to -time, but it isn't much use when you're buying, as many of the drives (because of the age of the tested ones) aren't on the market any longer, and they're Enterprise/Datacentre models anyway, not Consumer NAS Drives.

(FWIW, my experience over the years of WD has been slightly better than Seagate, but without more specific evidence, not enough to move me away from buying, like-for-like, on price. The current Seagate Ironwolf range integrate more easily with the NAS disk health management as well!)).

The suspect drive has been tested by independent and WD utilities. The independent ones all "suck their teeth" at it (It has trippped one of the S.M.A.R.T. triggers after all), WD says "move along, nothing to see here". Unfortunately, the NAS doesn't agree, and would continue to nag me to replace it - and No, I'm not feeling lucky.
Latest Forum Posts