minimising risk of full sync starwind HA following an outage

[kktwenty]

I have looked at:
https://knowledgebase.starwindsoftware. ... -blackout/
https://knowledgebase.starwindsoftware. ... xpectedly/
https://knowledgebase.starwindsoftware. ... may-start/

For a 3 HA HCI node + independent file witness setup using starwind free, I am trying to get a lab setup which will minimise the risk of a resync. Final setup will be a pair of 5Tb LUNs created in starwind in high availability, interlink sync channel will be either 40Gb or 100Gb (final spec to be decided), I cannot test a 10Tb resync over 40Gb or 100Gb at this time. The servers will present a volume to starwind that will be based on a "hardware cache disabled" RAID1 set of 6 SSDs so an all flash infrastructure with no hardware caching (an 8Gb cache is available and I may look at enabling for testing later). Looking at the above links I have come to the conclusion that to potentially avoid full resyncs (assuming all 3 nodes CAN come back online) I should do the following:

[*]Disable caching. Caching will most likely cause a full resync.
[*]Ensure that the primary node (if known) is booted first. Starwind will query the nodes + witness to determine "who is the most up to date" and "Fast" sync those changes to the partner nodes.
[*] Avoid the use of "Mark as synchronised" as much as possible as this forces a full resync to other partner nodes
[*] If (for example) node 1 was determined by starwind to have the most up to date data and node 1 is not available (i.e. node 1 has "died" and will not be restarted anytime soon) then a full resync is inevitable as you must mark either node 2 or node 3 as synchronised manually (thus triggering a full resync)

am I correct in thinking that starwind marks LUNs as read-only when performing a full sync but allows writes when performing a "Fast" sync?

[yaroslav (staff)]

Thank you for your questions but I believe there is some sort of misunderstanding here.
A full synchronization can start due to reasons outlined here https://knowledgebase.starwindsoftware. ... may-start/

If HA device was configured to use write-back cache and its flush did not complete correctly upon StarWind service restart, or one of the servers was turned off not correctly (i.e. hard reset, power outage, service interruption etc);

If the write-back cache is configured for a HA device, terminating StarWindService, as well as not letting cache flush, will drive full sync. Again, it affects only that device.

If the errors were detected during the writes on the disk;

E.g., storage malfunctions.

If partner, that should be the source of synchronization has the state other than “Synchronized”;
If initial synchronization was interrupted due to any reason;

Say, not letting synchronization finish (e.g., restart mishandle).

If the Partner that should be the source of synchronization gained “Synchronized” state after initiating “Mark as synchronized” command;
If StarWind service was turned off on all servers.

These 2 are connected.

Restarting nodes one by one results in fast sync. See more at https://knowledgebase.starwindsoftware. ... installed/.
Also check on the Maintenance Mode feature https://www.starwindsoftware.com/help/M ... eMode.html
Now, let me answer your questions.

This does not make much sense. If there is a power outage they both go down at the same time.

Not exactly. Write back cache causes full sync only if service is terminated on one node. In case of a power outage write-back cache might be a reason why you end up in a mutual not synchronized state for both devices.

[*]Ensure that the primary node (if known) is booted first. Starwind will query the nodes + witness to determine "who is the most up to date" and "Fast" sync those changes to the partner nodes.

No, for now, fast sync will never be the case if service is terminated/stopped on both nodes due to, say, shutdown or restart. We are working on eliminating full sync in case of an outage. Stay tuned!
On top of that, this does not make much sense. If there is a power outage they both go down at the same time.

[*] Avoid the use of "Mark as synchronised" as much as possible as this forces a full resync to other partner nodes

No. Marking as synchronized, is not what causes full synchronization. Marking as synchronized is what actually makes StarWind HA devices available again over iSCSI by telling which side to synchronize from if service cannot decide on it automatically (e.g., blackout or restart mishandle).

[*] If (for example) node 1 was determined by starwind to have the most up to date data and node 1 is not available (i.e. node 1 has "died" and will not be restarted anytime soon) then a full resync is inevitable as you must mark either node 2 or node 3 as synchronised manually (thus triggering a full resync)

Yes, but again, full synchronization starts not because of pressing the button but since there is no other data but node 2 and 3.

[kktwenty]

Thank you for taking time to reply. With regards to power, I have a rather nice opportunity. I have two buildings that are connected with a pair of fibre (40Gb) and a pair of copper (10gb). The runs are just short of 70m. They have independent power, and by independent I mean one building gets its supply from one substation and one from another (It was built after a local set of houses were constructed). There have been occasions (during a local flooding that affected the substation but not the site) whereby one building had no power and the secondary building did! During this time the cluster in site 2 effectively took over running of site 2 whilst the entirety of site 1 was out of action. One of my potential solutions is to have a two/three node HA direct connect with a fibre link NIC to NIC (ironically the 1gb heartbeat would be via vlan on a 10Gb switched network), they would retain independent source power (+UPS of course) though.

I have a lab environment set up for evaluation now and am going through various "force power off the VM" to see how it behaves whilst a third server connects to the iscsi targets doing "something". Since my hosts will be all flash, based on your comments the best thing I can do at the moment is to keep caching off, ensure I have at least one direct connection from host to host(s) - it seems best that the sync is direct connection as that can handle sync and heartbeat, try to at least have a shutdown script tied to UPS commands to at least try to shutdown gracefully. I am working on the last part for my testing.

Again, thanks for the info, it seems im on the right track with my testing. I had misunderstood that during a full resync ALL nodes were unavailable. This was not correct, only nodes that are SYNCHRONISED have available iscsi targets, any UNSYNCHRONISED nodes will not have available iscsi targets. This means that in the case where all nodes are unsynchronised (i.e. write back caching is enabled and all nodes are powered off at the same time), once "I" decide to mark one as synchronised, this node is available as an iscsi target and thus the cluster will run from this target (whilst the other nodes are full synchronised from it).

My final test will be a 7Tb full sync following a forced power off and manual "mark as synchronised" to see how long this will take, I need to do a little bit of work first with an old QNAP NAS to make this happen in the lab.

[yaroslav (staff)]

You are welcome!

I have two buildings that are connected with a pair of fibre (40Gb) and a pair of copper (10gb)

Please note that it is important to eliminate the single point of failure. In other words, these cables should go through different cable channels. If switches are used, make sure to have at least a pair of switches in each location.
If networking is SPoF, please consider using node majority.

one building gets its supply from one substation and one from another (It was built after a local set of houses were constructed).

That's actually quite cool! The thing is that having an unexpected shutdown will trigger full sync anyway if the Write-Back cache is configured. Without cache, there should be no full synchronization if one side goes dark. It does not apply to your case, I just want these thoughts to be here for other users

One of my potential solutions is to have a two/three node HA direct connect with a fibre link NIC to NIC (ironically the 1gb heartbeat would be via vlan on a 10Gb switched network), they would retain independent source power (+UPS of course) though.

Great!

I have a lab environment set up for evaluation now and am going through various "force power off the VM"

StarWind VSAN features active-active replication. If one server goes down, storage is available from another server. Please note that StarWind VSAN handles only storage high availability while moving the VMs to another server is a part of the cluster functioning.

Good to know that my long-post was helpful Please note that you are always welcome to contact us directly for some advice/help at support@starwind.com until this system is covered under TRIAL (i.e., GUI-enabled evaluation license). If this setup is going with VSAN FREE eventually, see you on Forum!