Automatic HA device synchronization after the both nodes...

[danisoto]

Hi,

One suggestion related to the "Automatic HA device synchronization after the both nodes were down" functionality added in version 5.8:

I'm searching for a setup for a PARANOID system without any user intervention at all. The HA option in the StarWind SAN software seems to be a good candidate, but the current version needs an administrator intervention after a power failure.

I suggest a "simple" improvement:

* Add two new parameters to the "Auto synchronization after failure" option when creating a "High Availability device":

• - "Auto-enable primary delay when partner is online": Seconds after secondary heartbeat channel is detected and both nodes are out of sync. If the timer is expired the primary goes "Synchronized" AUTOMATICALLY !!!
  
  - "Auto-enable primary delay when partner is offline": Seconds after service starts and secondary isn't detected. If the timer is expired the primary goes also "Synchronized" AUTOMATICALLY.

I understand the implications over data corruption of these options, but with a full tolerant filesystem over the shared iSCSI target I prefer to have the option to auto-enable the system than a user intervention. And please, remember that this will be an optional parameter: no one loses data if don't enable this!

My suggestion is set 5 minutes for first timer (secondary partner online) and 15 minutes for the second one (secondary not available).

Will be possible to add this to the next version, please?
Regards!

[Anatoly (staff)]

Hello danisoto,

- "Auto-enable primary delay when partner is online": Seconds after secondary heartbeat channel is detected and both nodes are out of sync. If the timer is expired the primary goes "Synchronized" AUTOMATICALLY !!!

- "Auto-enable primary delay when partner is offline": Seconds after service starts and secondary isn't detected. If the timer is expired the primary goes also "Synchronized" AUTOMATICALLY.

Actually in 5.8 we have "Auto-synchronization after both nodes failure" feature. That means that mostly synchronization will be initiated automatically after both nodes failure, and as soon as it will happen one of the StarWind nodes (the one with the most recent data) will be able for client connections.

[danisoto]

Hi Anatoly,

Thank you for your fast response! Nevertheless, I feel that you don't understand my suggestion. Le me to explain:

The topic of the message is "Automatic HA device synchronization after the both nodes..." in reference to the new function in version 5.8 (as I say at the beginning of the message). So, please, I suggest to go furthermore... I present a simple solution in case of TWO FAILURES. If you try to stop (power disconnection) of your two instances of the SAN, after the reboot both servers goes to the "not Synchronized" state. This the correct way to grant the data consistency, and the administrator needs to trigger the command "Full Synchronization (copy data to this device from partner)" in the node that you think has the worst data (Personally, I prefer a command for "Set this node as primary" in the other node that is more clear interface!). So, Why not make the option to AUTOMATICALLY set this state? Think you on this: If I label one node as PRIMARY, I can assume that this will be the valid node after TWO fails! I suggest to put a timer for optional auto-recovering WITHOUT user intervention. If you like to guarantee that the data is safe, you only need to DISABLE this timer (as the current behaviour).

Without this option (auto-enable primary in any scenario) your software don't have the requirements for our project, because we need to guarantee full automatic recovering after a power failure of both nodes (please, remember that on top of the exported volume we put a full redundant filesystem, and the possible data damage is acceptable).

If you need more information, I can provide to you.
Best regards!

[anton (staff)]

After both node down StarWind pick ups the one with the most recent version of your content to use as a synchronization source. Are you asking us to provide a switch to force ignore automatic check and just make one node pseudo-primary so after double fault it would always feed its data to its partner (does not matter who has the most recent content). Is it what you want? Just to confirm

Hi Anatoly,

Thank you for your fast response! Nevertheless, I feel that you don't understand my suggestion. Le me to explain:

The topic of the message is "Automatic HA device synchronization after the both nodes..." in reference to the new function in version 5.8 (as I say at the beginning of the message). So, please, I suggest to go furthermore... I present a simple solution in case of TWO FAILURES. If you try to stop (power disconnection) of your two instances of the SAN, after the reboot both servers goes to the "not Synchronized" state. This the correct way to grant the data consistency, and the administrator needs to trigger the command "Full Synchronization (copy data to this device from partner)" in the node that you think has the worst data (Personally, I prefer a command for "Set this node as primary" in the other node that is more clear interface!). So, Why not make the option to AUTOMATICALLY set this state? Think you on this: If I label one node as PRIMARY, I can assume that this will be the valid node after TWO fails! I suggest to put a timer for optional auto-recovering WITHOUT user intervention. If you like to guarantee that the data is safe, you only need to DISABLE this timer (as the current behaviour).

Without this option (auto-enable primary in any scenario) your software don't have the requirements for our project, because we need to guarantee full automatic recovering after a power failure of both nodes (please, remember that on top of the exported volume we put a full redundant filesystem, and the possible data damage is acceptable).

If you need more information, I can provide to you.
Best regards!

[danisoto]

After both node down StarWind pick ups the one with the most recent version of your content to use as a synchronization source. Are you asking us to provide a switch to force ignore automatic check and just make one node pseudo-primary so after double fault it would always feed its data to its partner (does not matter who has the most recent content). Is it what you want? Just to confirm

Hi Anatoly!

Yes and No!

Disable the current automatic check: no!
Set automatic primary online when the current automatic check fails after double fault: yes!

What is the problem? If both nodes crash, and it's IMPOSSIBLE to detect who has the more recent data, after reboot the server, the StarWind service don't start the target and both nodes stay in "not sync" state until forever. I need an option to auto-restart the service in this specific case! For sure, this will be started only after some time, and for this I suggest two different timeout triggers: one if both nodes are detected (in "no sync" state) and one if only primary is running (in "no sync" state).

Thank you for your time!

[danisoto]

Hi,

I try to explain in a more simple way:

My suggestion is "SET AUTOMATICALLY THE PRIMARY AFTER CHECK FAILS". After a double fail, and the nodes boot, both nodes start in a "out of sync" state and the service (iscsi exported) don't start at all! This state needs a user administrator to start the service forcing the sync of one node. My suggestion is: After several minutes, force the primary as "Synchronized".

[Anatoly (staff)]

danisoto,

That is actually pretty interesting idea! We will definatelly discuss its implementation in our software.

Thank you

[danisoto]

That is actually pretty interesting idea! We will definatelly discuss its implementation in our software.

Great!

[anton (staff)]

Stay tuned. Hope to have this one on V6 schedule

That is actually pretty interesting idea! We will definatelly discuss its implementation in our software.

Great!

[danisoto]

Stay tuned. Hope to have this one on V6 schedule

So, please, consider the two timeouts: the first is when both nodes are up, the second when only one is up and the other is down (or you fell that is down!).

[Anatoly (staff)]

Definatelly!

Thank you once again! We really appreciate your input:)

[Steve9R]

hmm i had a similar occurance to this occur the other week.

what happens then when you have HA running both have a hard crash, but because you run multipath and some servers are using the alt. path when you force bring up one node it causes massive issues because its not the most recent data for those using the alt path at the time ??

Steve

[danisoto]

hmm i had a similar occurance to this occur the other week.

what happens then when you have HA running both have a hard crash, but because you run multipath and some servers are using the alt. path when you force bring up one node it causes massive issues because its not the most recent data for those using the alt path at the time ??

Steve

Hi, set write-through caching to avoid this failure!
Or use a filesystem on top that can tolerate errors.

Regards.

[anton (staff)]

+1

Using huge write-back cache is allowed only if both storage is configured to HA and client is clustered as well. In an opposite case enormous amount of transactions
file system "thinks" are commited to disk are actually lost rendering file system layout broken, data lost and storage system useless waiting for restore from last backup.

That's why OP should re-build his cluster from the ground step-by-step:

1) Use StarWind in HA configurtion.

2) Make sure both storage nodes have working UPS modules.

3) Have his hypervisor (VMware) updated to at least dual head mode.

Everything referenced is not StarWind issue it's the way things work. Any storage system (including built-in RAID) are expected to have impact from power outage or writer crash.

hmm i had a similar occurance to this occur the other week.

what happens then when you have HA running both have a hard crash, but because you run multipath and some servers are using the alt. path when you force bring up one node it causes massive issues because its not the most recent data for those using the alt path at the time ??

Steve

Hi, set write-through caching to avoid this failure!
Or use a filesystem on top that can tolerate errors.

Regards.

[anton (staff)]

You're not expected to do what you're doing! You cannot bring one node up and use it for production. You need to bring up BOTH nodes, run a synchronization process between them and use full serviced cluster in production after that.

See it's only 50% probability (with 2 node config of course) you'll pick up yourself the node with the most recent version of your data.

hmm i had a similar occurance to this occur the other week.

what happens then when you have HA running both have a hard crash, but because you run multipath and some servers are using the alt. path when you force bring up one node it causes massive issues because its not the most recent data for those using the alt path at the time ??

Steve