iSCSI Service Unavailable - Node 3 Unsynchronized

[kristysimsx]

My issue is almost exactly like in https://forums.starwindsoftware.com/vie ... f=5&t=4792

However the difference is that I have a 3 node cluster. After a reboot the node came up with the iSCSI targets for the Witness and the Volume inactive. I tried all of the Microsoft debugging but when I tryto reconnect the target, it says iSCSI service unavailable. I have tried recreating the target and Starwinds KB https://knowledgebase.starwindsoftware. ... kb4019215/ but this did not help either.

ISCSI.jpg (98.62 KiB) Viewed 3518 times

This is the Starwind Console on the Node3

Node3-Starwind.jpg (163.46 KiB) Viewed 3518 times

If I follow Ivans' advice in the similar post, he says go to a known node with the "Most recent data" (this would be either Node 1 or Node 2 for me, and open the Starwind console select the unsynchronized node 3 (HAImage2 or the Witness HAImage3) I get this message. This freaks me out as this is a production cluster so I don't know what to do.

This is Node 2 clicking on the Node 3 connection and clicking the "Marked as Synchronized" on either one of the Volumes

Node2-Starwind-Message.jpg (144.51 KiB) Viewed 3518 times

[yaroslav (staff)]

After a reboot the node came up with the iSCSI targets for the Witness and the Volume inactive

Check the favorite targets tab.

Are devices on other nodes not synchronized?
If so, this article might help https://knowledgebase.starwindsoftware. ... -blackout/.
Also, it appears that you have access to GUI. Please contact StarWind Support support@starwind.com

[kristysimsx]

The other nodes are fine Node 1 can connect to itself and 2 and 2 can connect to itself and node 1 Node 3 is having the problem connecting to itself, 1 and 2


Favorites tab attached:

iSCSI Favorites.jpg (68.47 KiB) Viewed 3496 times

I read the article however Node 1 and 2 that are fine do not give the “Mark as Synchronized” option. Only Node 3 however

based on the warning I got I can't risk losing a production cluster (Hence posting here)

Thanks for the assistance

[yaroslav (staff)]

Nodes 1 and 2 are synchronized since they are presenting the storage.
Please restart the StarWindService on the not synchronized node and collect the logs using https://knowledgebase.starwindsoftware. ... collector/.
Also, I can see you have access to GUI. Please contact StarWind support at support@starwind.com use this thread as a reference.

[kristysimsx]

I have restarted the service and the entire node on multiple occasions. I don't have support anymore the cost to Bring them up to support would cost the same as a physical SAN

I am wondering now if the problem isn't Node 3 Here are the iSCSI connections on all three

Node 1: cluster_node_1.xxxxx

Node 2: 192.168.1.92

Node 3: 192.168.1.94

iSCSI-Node1.jpg (82.18 KiB) Viewed 3477 times

iSCSI-Node2.jpg (90.89 KiB) Viewed 3477 times

iSCSI-Node3.jpg (107.34 KiB) Viewed 3477 times

[yaroslav (staff)]

It's not about the iSCSI connection alone. Can I have the logs from 3 nodes, please?

[kristysimsx]

Sure, Uploaded here https://drive.google.com/file/d/1mE_8BV ... sp=sharing

[yaroslav (staff)]

Can I have the logs from the other 2 nodes, please?

[kristysimsx]

Sure

Node 2 https://drive.google.com/file/d/1-DIfVN ... sp=sharing

Node 1 https://drive.google.com/file/d/1oW-TTa ... sp=sharing

[yaroslav (staff)]

Greetings,

I noticed that StarWind VSAN is outdated in this system. Consider updating it as described here https://knowledgebase.starwindsoftware. ... d-version/. Get the up-to-date version from the release notes https://www.starwindsoftware.com/release-notes-build.
Also, I something that looks like a disk issue on node3 Severity: Warning, Category: Storage, MessageID: PDR16, Message: Predictive failure reported for Physical Disk 0:0:0 on Controller 0 at Connector 0. READ delays were the reason for a device going out fo sync.
Could you please check the underlying storage? Due to delays, StarWind VSAN can't synchronize.

Please let me know if you find anything on the underlying storage level.

[kristysimsx]

Storage is fine, however we have a planned maintenance window to replace the drive which is part of a RAID 1 and only used for the OS. The VSAN Array is fine and is a RAID 10 so even if the drive failed the VSAN would be okay so I am not sure the problem " Due to delays, StarWind VSAN can't synchronize." Are you saying that the whole sync problem is related to a failing disk in the OS? If so doesn't that defeat the purpose of a VSAN in the first place (That is would kill an entire Node over one impending device failure in an already redundant RAID 10 system?

The cost to upgrade exceeds the revenue to host (It is a small company) it has been running flawless for years as is, so they just want me to help get the re-sync to work.

[yaroslav (staff)]

Hi,

What I am trying to say is that StarWind VSAN stumbled into underlying storage delays and could not synchronize. It is not related to the OS drive, but to the underlying storage. The updated version introduced higher timeouts for underlying storage, hence, it may make the VSAN service synchronize. The service became tougher.
It does not defeat the purpose as you are still having 2 mirrors. Even in the 2-way mirror, with a node down the production is running.

If the update is done on your own, you can update it for free. You still can update as VSAN as currently, your system is acting as a 2-way mirror. It is the support contract that needs to be renewed; the product itself does not expire and you can switch to later versions even without an active support plan.
HINT: you might require the key change. Please contact support@starwind.com (use this thread and 742222 as your references). If you need a key during the update, it will be supplied faster I'd suggest starting with the affected node.

[kristysimsx]

Thanks for the help! doing the upgrade on all three nodes has triggered the sync. I'll let you know how it goes. I didn't need a new key either

[yaroslav (staff)]

Great news! Glad to know that
Keep me posted.