The hyper-v clustering side I have sorted. That is already in place and a mature solution that has been tested and used with power failure. This is our CURRENT setup:
We have two physical buildings, the primary building is classed as our "main" building. Internet connection enters here and has the bulk of our networking. The secondary building is a smaller building connected by a redundant pair of 10Gb lines. To mitigate the risk of loss due to fire, flood or catastrophic loss of the main server room it was decided to have a resilient system between the two buildings. Currently we use MS stretch cluster, this has a pair of servers connected to a SAN in primary building and a pair of servers connected to a SAN in the second building. A NAS physically located in the primary building acts as a quorum. The nodes in the secondary building do NOT have a vote in the cluster. This is how the system works now:
Scenario 1 - node in primary site fails. Quorum + second node decide that second node will now host all the VMs until failed node is returned to use.
Scenario 2 - node in secondary site fails. Nothing happens to the running of the cluster other than an error that a secondary node has failed.
Scenario 3 - storage fails in primary site. Redirected I/O to the storage in the secondary site takes place. VMs remain running on primary nodes using redirected IO. Once the primary SAN is back and running there will be heavy network usage as the data is re-synced back from secondary site to the primary site - this is also whilst redirected I/O is taking place (this IS slow!)
Scenario 4 - power failure in secondary site. Nothing happens to the running of the cluster. Heavy IO use when the power is returned and a resync takes place from primary to secondary.
Scenario 5 - power failure in primary site. The cluster shuts down totally, the secondary site needs to be issued with powershell commands to come online, the storage is accessible and not marked as "degraded", the cluster can be brought online with a set of three powershell commaneds: Votes are added manually to the secondary site so that when the primary nodes are brought up they will defer to the secondary site as authoritative. Primary site is manually started up and heavy resync from secondary to primary takes place, primary site acknowledges secondary site as authoritive since they have votes now. communication between secondary and primary MUST work before primary site is rebooted to avoid split brain (as quorum + two nodes on primary would be enough to continue running independently!).
There is no internet in secondary site so a cloud quorum cannot be used. Since scenario 5 deals with a network failure between the two sites there would be no "middle denominator" visible to both sites. Power is independent to both sites and they are situated 100m apart physically. I have experienced all 5 of these scenarios with both testing and in practice.
It is scenario 5 that I am most hesitant with starwind. My new proposal is for either S2D or Starwind to operate with 3 nodes. Two nodes in the Primary site and one in the secondary. From my research, S2D can be configured exactly the same way as hyper-v cluster: I can add votes to the primary site nodes (+ quorum) and take a vote away from the secondary node. I have not tested this as I am at the planning stage. I would like to see if starwind can be set as similar. This is my PROPOSED setup:
Scenario 6 - starwind+compute nodes 1 and 2 live in primary site and have visibility of a quorum NAS. Secondary site has starwind+compute node 3. Network visibility between primary and secondary site is lost - this could be due to power loss, someone cutting through my fibre, fire in primary site. I know already that I can set my cluster to have "no votes" therefore the cluster will need to be brought up manually,
however I dont know if the starwind node 3 can be brought online for the hyperconverged cluster on node 3 to be used or does starwind node 3 list itself as degraded. Once visibility of nodes 1 & 2 (plus the quorum) come back up, how do I let starwind know that node 3 has been authoritive whilst nodes 1,2 + quorum have been "down"? I dont want starwind 1,2+quorum deciding that "they" are authoritive whilst node 3 has been running the cluster+storage during the failure.
In hyper-v cluster (in this situation) the act of adding votes to my secondary site is enough to convince the cluster that the secondary site is authoritive - even though "technically" the primary site comes back with votes + quorum.
Nodes 1,2,3 will be set as synchronous HA of course. Incidentally backups are stored in primary and secondary sites.
EDIT: Having read
https://www.starwindsoftware.com/blog/w ... o-avoid-it what I am looking for is witness node whereby node 1,node2, witness node get a vote. node3 does not get a vote. Therefore in a normal network partition (link between primary and secondary is cut so it is just a communication issue) the primary site will correctly say that the secondary site node 3 is "not synchronised". However, if nodes 1 and 2 are brought down by power failure (along with any suitable witness or quorum), then I would like to use node3 as the cluster storage (and bring up the cluster manually on node3).
I did see this caveat at the bottom of the article:
In case if an HA device consists of three storage nodes or two storage nodes and one witness node, only one node can be disabled. In case of failure of two nodes, the third node will also shut down.
which leads me to believe this is not only impossible with starwind but also makes a 3 node HA pointless as it appears starwind will not operate with 2 HA nodes out of 3 HA nodes failed? Or have I read this incorrectly?