Cluster Storage offline, can't bring online

[Mareo]

Greetings,

This is my first time posting here.

We've encountered an issue twice now where the Starwind services could not be brought back online. The service falls out of sync, preventing me from bringing the VM, CSV, and Witness online.

I've seen similar issues discussed here multiple times, but despite following the steps closely, I couldn't resolve the problem. I think I need specific guidance for my case. Could you help me interpret the logs? If needed, I can upload them (which I believe will be necessary).

The problem starts when we connect to the servers and find them they were shutdown. Upon launching Cluster Failover Management, both servers appear disconnected from the Cluster. Consequently, the VM goes offline along with the Storages and said nodes. Attempting to bring the VM online fails due to a storage error. Both the CSV and the Witness are offline, and efforts to bring them online result in errors indicating they are not connected to the node.

However, both nodes are online.

Furthermore, when enabling Starwind and connecting to the servers, both show as "not synchronized." Running PowerShell scripts also fails, with errors such as "200 Failed: can't find partner for sync..."

Additionally, the iSCSI initiators are stuck in a reconnecting state.

Note: We are using the free version of Starwind. The operating system is Windows Server 2022 Standard, and we are using HyperV.

I'm available to provide any additional information you might need.

Thank you in advance, and I look forward to your assistance.

Kind regards,
Mario

[yaroslav (staff)]

Thanks for your email. Did you have a power outage? Do your devices have a write-back cache on them?

Additionally, the iSCSI initiators are stuck in a reconnecting state.

Are devices from both sides stuck in reconnecting?
Check this post out viewtopic.php?f=5&t=6779&p=36805&hilit= ... %3E#p36805.

There might be another problem: The domain controller is in the cluster. As a result, the cluster cannot start because DCs are down and DCs do not start as the cluster is down. See more on DC placement https://knowledgebase.starwindsoftware. ... san-usage/.
Force-start the cluster as described here https://learn.microsoft.com/en-us/sql/s ... rver-ver16.

[Mareo]

Hello,

thank you for the reply.

Kindly read the EDIT first.

I will try these things first thing possible.

However, I do have an additional question. If we only remain on one node, can I recover the VM from HyperV? At the current state, we are only on one node and we DO NOT have access to the Cluster to retrieve the VM. VM is in an offline state and we cant access the Cluster folder since there is currently no cluster due to there being only one node. Is that affected by Starwind or is there no correlation between the two and I just need to find a way(to get the VM back)?

Thanks in advance and kind regards,
Mario

EDIT: Please, disregard this post. I mixed up the nodes.

[yaroslav (staff)]

Yes, you can continue running with only one node marked as synchronized. Be careful and check your data before resuming the other node.
Also, considre removing cache once things get to norm again. And, last but not least, move DCs out of the cluster and cluster shared volumes.
Please note that the failover cluster is non-StarWind component. I cannot predict if it can be force started.

[Mareo]

I just saw your reply. I edited the post your replied to. Regardless, thank you for your assistance.

Furthermore, I forgot to mention that we do not have the DCs in the Cluster. They are standalone VMs. Does this change the situation?

Kind regards,
Mario

[yaroslav (staff)]

Thanks for your update.
StarWind VSAN does not need a DC for its operation. Mark one StrarWind node as synchronized manually and try bringing up the cluster. Once the cluster starts and StarWind devices are marked as synchronized on one node, you should be able to access the CSVs.

[Mareo]

Referring to a post you quoted regarding the sync status:

Try marking sync_status 1 only the first occurance. Set the latter to 0.
Make sure to edit the files while StarWindService is stopped.
Last but not least, remove write back cache: should make such incidents less frequent.

I'm kinda lost in what file do I need to change the sync status? HA file on the first node is for the second node(target) and vice versa. Does the sync status refer to the target node above it? If I want to set sync status 1 for the second node in which HA file do I edit the sync status?

[yaroslav (staff)]

Hi,

The first one is for the local device. every other occurrence is for the respective partner.
You need to first mark one node as synchronized and check the data consistency. If the data is OK make sure to mark the device with the respective target name in the config files with <sync_status>1</sync_status>

[Mareo]

Hi,

firstly, I would like to thank you for your support.

Secondly, I got it working! I followed the links and instructions you posted, edited the HA images with steps in between like stopping/starting services, then I used the GetHASyncStatus file. It took me a few tries, but it finally synced.

The resources are back online and the VM is up and running as well.

However, can you kindly confirm would the same work for a similar issue, but where everything is online. The resources and the VM is online and everything is working. Only the HAImages in Starwind are showing unsync status. I take it I wont be needing to edit the HA images? Maybe just do the GetHASyncStatus?

Thanks in advance and kind regards,
Mario


EDIT: Sorry, I mixed it up.. The second issue in question that one node has the CSV and the witness error images as in "Header not found". But for the first node the the second node is down/unsynchronized.

[yaroslav (staff)]

Secondly, I got it working! I followed the links and instructions you posted, edited the HA images with steps in between like stopping/starting services, then I used the GetHASyncStatus file. It took me a few tries, but it finally synced.

Sorry, I don't quite follow. Is the problem still there or is it a different system?
Please check which StarWind VSAN version you are running first. In StarWind Management Console, select a server -> Configuration -> Register and see the version there.

[Mareo]

Secondly, I got it working! I followed the links and instructions you posted, edited the HA images with steps in between like stopping/starting services, then I used the GetHASyncStatus file. It took me a few tries, but it finally synced.

Sorry, I don't quite follow. Is the problem still there or is it a different system?
Please check which StarWind VSAN version you are running first. In StarWind Management Console, select a server -> Configuration -> Register and see the version there.

This is the same system.

But, the second part of the email is about a different system. I just remembered that I copied the witness and CSV files to the second node. I did find a thread about a similar issue, unless I mixed it up and got it wrong. So, naturally, my question is - Can I copy the images from one node to the other, edit the images to fit the second node(change target name, sync status, etc.) and get it working again?

[yaroslav (staff)]

Hi,

Theoretically yes. But that introduces a room for mistake and data is likely to be inconsistent as the new data arrives and gets altere during copy. Furthermore, iI would still liketo let full synchronization running.
If there's volume recreation, please run replication partner removal on the only active node. Or alter the _HA.swdsk to remove the replication partner (requires replication partner removal).then, run addreplicationpartner script.
Let me know if you would like to learn more details.