Free VSan Recovery, Hash Sync, iSER

[LearnVSan]

Had the misfortune of a total VSan crash (service crash) after changing network cards. Since both nodes crashed, it was not possible to enable synchronization no matter how many posts I read (here and elsewhere) - spent hours trying all kinds of things and nothing would start synchronization.

1. It is *not* enough to mark one node as synchronized and other not synchronized (in xxx_HA.swdsk)

Code: Select all

<sync_status>1</sync_status>

and

Code: Select all

<sync_status>0</sync_status>

for the other node.

You also need to set

Code: Select all

... shut="true"... 

for the node you marked as synchronized - I assume this means "it has shut down ok"? - otherwise the sync will not start whatever you do. For example:

Code: Select all

<node id="1" name="HAImage" shut="true" active="true" flags="0">

Once I did this - sync started as soon as I started the VSan service.

2. There seem to be 2 types of full sync - "Full" that copies everything across and "Full Hash" only for StorageSpaces disks that reads both nodes and compares hashes.

Can this "full hash" is be used for other devices, especially SSDs? It would help a lot with SSD wear not to just copy a whole SSD onto another when it could just have compared the hashes?

Especially since it copies entire image size even if file system inside is 20% used and image size close to SSD size. I assume when physical disk is HA'd it does not copy/hash the entire hard disk but just what's used.

3. Crash happened after I enabled "iSER" as these new cards have RoCE - I see the recommandation is to disable it as it did not take off?

But even if it were used by VSAN since that is both target and initiator iSER would still help internally even if external initiators don't use iSER?

4. Regarding [2], seems even trial licence does not enable HA for physical disks. Personally I'd rather have that and not .img support - especially when crashes, synchronization problems, etc. occur.

Worst-case you just want to access the data and dealing with .img on top of yet another file system - 3 file systems (host, img, vm).

I hope this helps.

[yaroslav (staff)]

Can this "full hash" is be used for other devices, especially SSDs? It would help a lot with SSD wear not to just copy a whole SSD onto another when it could just have compared the hashes?

Full hash sync is happening only for Storage Spaces.
Speaking of how to mark the node as synchronized. Use SyncHaDevice.ps1, comment $device.Synchronize([SwHaSyncType]::SW_HA_SYNC_FULL, "") and uncomment #$device.MarkAsSynchronized()

3. Crash happened after I enabled "iSER" as these new cards have RoCE - I see the recommandation is to disable it as it did not take off?

That's the known issue for 19551. Don't use iSER, please.

Also, I think you might need to remove write-back caching from your HA devices.

4. Regarding [2], seems even trial licence does not enable HA for physical disks. Personally I'd rather have that and not .img support - especially when crashes, synchronization problems, etc. occur.

I think the issue is not with StarWind VSAN here but with the Cluster File system locking access AND VSAN entering a mutual not synchronized.
Remove Write-Back cache, and that should happen less often. If node majority is used, make sure that the node never gets isolated.
The Physical disk mapping is a legacy thing that does not get enough light these days, also it is not available for HA devices.

[LearnVSan]

Thanks for replying.

Unfortunately I did not know that needed commenting out. I ran that script many times on either node and naturally they were both "unsynchronised". Perhaps there should be a flag/option for that script to force this marking as it was not obvious how to fix it. Only discovered that flag comparing with older/backup .swdsk files.

iSER: That's a pity. It seemed a good idea even just internally for the SYNC interface? SmbDirect seems to work as far as the TestRDMA Microsoft script thinks - tested both via iSCSI and SYNC networks.

Hash Sync: I really think this should be the default; the code path is already written and used for StorageSpaces can't this be enabled for SSDs too or perhaps just in all cases? Why wear out SSDs copying TB of data across when you can just compare the hashes?

VSan service crashed on both nodes once I enabled iSER on both nodes - you can have the dump files (.mdmp). I renamed "iSER_DM.dll.bak" as you recommended in another thread and seems stable. Just finished sync on HDD array after 3 days.

Thanks for script tip - next time will try that to force sync.

[yaroslav (staff)]

You are always welcome! I am happy to help.
Thanks for your comments, let me try addressing them.

Perhaps there should be a flag/option for that script to force this marking as it was not obvious how to fix it. Only discovered that flag comparing with older/backup .swdsk files.

As there's the script, we decided to leave things as is.

That's a pity. It seemed a good idea even just internally for the SYNC interface?

The problem with iSER is that it rolls back on service restart to TCP. The only solution is to restart the service.
With this build, though, VSAN just crashes if iSER is used. We are working on it.

I really think this should be the default; the code path is already written and used for StorageSpaces can't this be enabled for SSDs too or perhaps just in all cases?

As a person who manages StarWind HA on a daily basis, I'd say stay away from full hash sync. It just takes too long and has a great chance of overlapping with other activities and restarting due to high i/o from other processes (e.g., production).
It is there to address MS Storage Spaces causing BSOD under high i/o. Full hash still involves huge data transfers, yet slower.