Information you provided in your original post in this thread does not coincide with the logs.
According to you, the issue manifested on Saturday (i.e. January, 12), as the nodes went down after power outage. Yet, according to the logs, the nodes were able to get all disks synchronized on Sunday, Jan 13:
Code: Select all
788 HyperCluster1.electro-core.com 1948230 Information High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-downloads, current Node Synchronization complete, Synchronizer is Partner Node iqn.2008-08.com.starwindsoftware:hypercluster2-downloads StarWindService 1/13/2019 11:37:30 AM
773 HyperCluster1.electro-core.com 1948229 Information High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-downloads, current Node State has changed to "Synchronized" StarWindService 1/13/2019 11:37:30 AM
787 HyperCluster1.electro-core.com 1948228 Information High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-downloads, current Node Synchronization started, Synchronizer is Partner Node iqn.2008-08.com.starwindsoftware:hypercluster2-downloads StarWindService 1/13/2019 11:37:28 AM
774 HyperCluster1.electro-core.com 1948227 Warning High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-downloads, current Node State has changed to "Synchronizing" StarWindService 1/13/2019 11:37:28 AM
788 HyperCluster1.electro-core.com 1948226 Information High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-fileserverstorage, current Node Synchronization complete, Synchronizer is Partner Node iqn.2008-08.com.starwindsoftware:hypercluster2-fileserverstorage StarWindService 1/13/2019 11:37:27 AM
773 HyperCluster1.electro-core.com 1948225 Information High Availability Device iqn.2008-08.com.starwindsoftware:hypercluster1.electro-core.com-fileserverstorage, current Node State has changed to "Synchronized" StarWindService 1/13/2019 11:37:27 AM
902 HyperCluster1.electro-core.com 1948224 0 "The Software Protection service has started.
6.1.7601.17514" Software Protection Platform Service 1/13/2019 11:36:46 AM
Finally, the devices went out of sync on the same day, Jan 13 at 12:56:16 PM, when the following happened:
Code: Select all
"The process Explorer.EXE has initiated the restart of computer HYPERCLUSTER1 on behalf of user ELECTRO-CORE\Norm for the following reason: Application: Maintenance (Planned)
Reason Code: 0x84040001
Shutdown Type: restart
Comment: "
Also, there was an unexpected shutdown on node 1:
Code: Select all
The previous system shutdown at 1:35:12 PM on 1/13/2019 was unexpected.
This looks pretty much like the event you initially meant. Am I right?
Anyway, the present status is as follows - your devices on node 1 are non-active because of the HA header files missing for three out of four disks on node 1. The one for Storage1 is still there, but its structure is totally corrupted, and thus the file is not usable at all. Unfortunately, StarWind logs cover only the time starting from 1/13 15:59:09.821 (overwritten by log rotation), so we are not really able to define what exactly happened there. Check your Windows Secutiry log for event 4660 related to Witness_HA.swdsk, Storage1_HA.swdsk, FileServerStorage_HA.swdsk and Storage2_HA.swdsk on node 1 after Jan 13, 12:56:16 PM for more information on what or who deleted the files, but only on condition the file system audit had been configured there.
In the current situation, I recommend you the following:
1. Force remove all targets on node 1.
2. Remove all StarWind related folders with their files from the D drive on node 1.
3. For each of the devices on node 2, select Replication Manager and delete the non-existing replicas.
4. For each of the devices on node 2, create replica to node 1 using the appropriate sync link and the two heartbeat links.
5. In the iSCSI initiators on both nodes, connect the newly appeared targets.
After connectivity to storage is restored from both nodes and the paths to storage are redundant, I would recommend you updating your StarWind installation to the latest build available at our website, as you keep using a pretty outdated build.