"Device is not synchronized"

[Ardie]

Configuraiton:
Two Hyper-V 2016 nodes, A and B.
Three HA images, CSV1 (SSD), CSV2 (HDD), and Witness.

A - Management - 192.168.0.1
A - ISCSI - 10.1.2.1
A - Sync - 10.2.2.1

B = Management - 192.168.0.2
B - ISCSI - 10.1.2.2
B - Sync - 10.2.2.2

Node A has all HAImage objects showing Synced and everything is working.
Node B has the first two HAImage objects showing Synces, but CSV2 is not.

B has just been through a restart (clean restart, no hard shutdown or power loss).

Witness and CSV1 on Node B are normal, however CSV2 on B now says:

!: Device is not synchronized
X: Synchronization channel '10.2.2.1:3260' on 'iqn.2008-08.com.starwindsoftware:192.168.0.1-csv2' is down!
X: Heartbeat channel '192.168.0.1:3260' on 'iqn.2008-08.com.starwindsoftware:192.168.0.1-csv2' is down!
X: Heartbeat channel '10.2.1.1:3260' on 'iqn.2008-08.com.starwindsoftware:192.168.0.1-csv2' is down!
X: Partner 'iqn.2008-08.com.starwindsoftware:192.168.0.1-csv2' is not synchronized

When I check Replication Manager for HAImage3 it shows "Connection Status Down". At the bottom of the window the IP addresses for Synchronization Channel and Heartbeat Channel all have a red X at the end of them.
When I check Replication manager for HAImage1 and HAImage2 everything is connected and the IP addresses have a blue check at the end.

I can ping from both hosts to all three interfaces on the other host.

iscsicpl shows the -csv2 target as "Reconnecting" for both the local and remote nodes on node B.

Why does this target show as disconnected when it connects the same way as the others? How do I get it to reconnect? I tried removing/readding one of the interface addresses in Replication Node Interfaces with no change.

Not sure what to check from here, any ideas?

Thanks.

[Boris (staff)]

It looks like your physical storage under CSV2 had some trouble at the moment of starting, and thus it marked that node as not synchronized by the device priority, but for some reason it has not restored it automatically. The first troubleshooting step here would be forcing synchronization of CSV2 from the StarWind Management Console. If that does not help, restart the StarWind VSAN service on the node where CSV2 is in the non-synchronized state. Additionally, check the Windows System logs and the storage controller ones for any unusual events related to storage health.
Report the result.

[Ardie]

The Synchronization option is greyed out in the console, I assume it is simply marked as "Not synchronized" due to the fact it says it has no connection on any of the IP addresses to NodeA, despite the other two images being connected the same way.

I have restarted the StarWind VSAN service on NodeB which results in the same connected status for HAImage1 and HAImage2, but the same not connected error messages on HAImage3. The host has also been restarted twice with no change.

There are no storage related errors in the Windows event logs and the RAID controller reports no errors.

The other strange thing is that Replication Manager for HAImage3 on NodeA shows all of the connection statuses as active to NodeB, they only show as disconnected from NodeB to NodeA.

Should I try to remove the replica on NodeB (the one that says it cant connect to NodeA) and set it up again?

[Boris (staff)]

You can try removing the replica and recreating it. That should ultimately solve it.