StarWind Software

iSCSI, FCoE, AoE solutions for SMBs and Corporations. VTL, CDP and other advanced features.
https://forums.starwindsoftware.com/

Hyper-V 2 node cluster consistently restarts full sync

https://forums.starwindsoftware.com/viewtopic.php?f=5&t=7348

Page 1 of 3

Hyper-V 2 node cluster consistently restarts full sync

Posted: Thu Dec 11, 2025 6:06 pm
by Electrum
Hello,

I have a 2 node Hyper-V cluster running vSAN. Host 01 is the source of truth for the disks. I am able to define iSCSI connections on Host01 utilizing the loopback.

Host 02 is able to define iSCSI connections using it's 25G NIC(s) to Host01. However, Host01 isn't able to define Host02 as an iSCSI source, and Host02 is unable to define it's loopbacks as an iSCSI source. In either case, the service is "unavailable"

If I manually synchronize through the GUI, they will work fine. For a short amount of time, both will be marked as "good" for a short file, until Host02 reports it is no longer synchronized, and the process begins again.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Thu Dec 11, 2025 6:24 pm
by yaroslav (staff)
StarWind VSAN features an active-active mirror. There's no primary or secondary host. See more on priorities viewtopic.php?t=5731.

If synchronization drops happen, please check the logs. Check your netowrks/storage. Review the logs and see if there are any delays, network interruptions.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Thu Dec 11, 2025 6:58 pm
by Electrum
Hello,

I have host02 set with a priority of "second" for all HAImages.

Here is a snippet of what I am seeing:

Code: Select all

12/11 13:27:57.161353 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full' 
12/11 13:49:39.331194 5848 Common: *** iscsi_tcp_dispatch: request execution timed out (9953 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144717, maxcmdsn = 8144964.
12/11 13:49:39.331366 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144710, itt = 8150106)
12/11 13:49:39.331397 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144711, itt = 8150107)
12/11 13:49:39.331426 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7ECBC8C88 has failed due to unidentified error.
12/11 13:49:39.331441 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.331464 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC44BE90 synchronization task has been failed!
12/11 13:49:39.331498 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F3693C88 has failed due to unidentified error.
12/11 13:49:39.331512 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.331533 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC44EC90 synchronization task has been failed!
12/11 13:49:39.331557 32b8 func: >>> CHAPartnerNode::SetSyncStatus
12/11 13:49:39.331614 32b8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/11 13:49:39.331632 32b8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 3 from 2. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
12/11 13:49:39.331647 32b8 func: >>> HANode::reportEvent
12/11 13:49:39.331676 32b8 func: >>> iScsiServer::sendNotification
12/11 13:49:39.331690 32b8 func: >>> CEventDataBase::AddRecord
12/11 13:49:39.331701 32b8 EventDB: CEventDataBase::AddRecord: Code 771, severity 3, additional strings: 2
12/11 13:49:39.331923 32b8 func: <<< CEventDataBase::AddRecord
12/11 13:49:39.331938 32b8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 771).
12/11 13:49:39.333107 32b8 Srv: iScsiServer::sendNotification: Send reaction finish.
12/11 13:49:39.333159 32b8 func: <<< iScsiServer::sendNotification
12/11 13:49:39.333183 32b8 func: <<< HANode::reportEvent
12/11 13:49:39.333196 32b8 func: <<< CHAPartnerNode::SetSyncStatus
12/11 13:49:39.334893 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144709, itt = 8150105)
12/11 13:49:39.334953 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144710, itt = 8150106)
12/11 13:49:39.334975 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144711, itt = 8150107)
12/11 13:49:39.334999 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F00A9C88 has failed due to unidentified error.
12/11 13:49:39.335021 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.335037 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29BDC0 synchronization task has been failed!
12/11 13:49:39.335077 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F00AA808 has failed due to unidentified error.
12/11 13:49:39.335097 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.335111 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29D1E0 synchronization task has been failed!
12/11 13:49:39.335147 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F00A9988 has failed due to unidentified error.
12/11 13:49:39.335180 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.335202 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29C380 synchronization task has been failed!
12/11 13:49:39.347802 5848 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144717, maxcmdsn = 8144964.
12/11 13:49:39.347864 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144712, itt = 8150108)
12/11 13:49:39.347882 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144713, itt = 8150109)
12/11 13:49:39.347897 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144714, itt = 8150110)
12/11 13:49:39.347910 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144715, itt = 8150111)
12/11 13:49:39.347929 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EAA52C48 has failed due to unidentified error.
12/11 13:49:39.347943 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.347965 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC44DB50 synchronization task has been failed!
12/11 13:49:39.347995 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7ED056098 has failed due to unidentified error.
12/11 13:49:39.348009 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.348029 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29C0A0 synchronization task has been failed!
12/11 13:49:39.348051 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E95539E8 has failed due to unidentified error.
12/11 13:49:39.348065 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.348085 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29B800 synchronization task has been failed!
12/11 13:49:39.348106 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7ECBC9188 has failed due to unidentified error.
12/11 13:49:39.348119 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.348139 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC44F530 synchronization task has been failed!
12/11 13:49:39.351801 a90 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144716, maxcmdsn = 8144963.
12/11 13:49:39.351854 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144712, itt = 8150108)
12/11 13:49:39.351880 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144713, itt = 8150109)
12/11 13:49:39.351901 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144714, itt = 8150110)
12/11 13:49:39.351923 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144715, itt = 8150111)
12/11 13:49:39.351956 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F3693E08 has failed due to unidentified error.
12/11 13:49:39.351986 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.352005 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29CF00 synchronization task has been failed!
12/11 13:49:39.352042 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E9553F68 has failed due to unidentified error.
12/11 13:49:39.352073 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.352101 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29C940 synchronization task has been failed!
12/11 13:49:39.352136 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7F3693508 has failed due to unidentified error.
12/11 13:49:39.352155 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.352169 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29C660 synchronization task has been failed!
12/11 13:49:39.352196 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E95531E8 has failed due to unidentified error.
12/11 13:49:39.352216 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.352230 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29B520 synchronization task has been failed!
12/11 13:49:39.363674 5848 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144717, maxcmdsn = 8144964.
12/11 13:49:39.363737 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144716, itt = 8150112)
12/11 13:49:39.363758 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EC56AB98 has failed due to unidentified error.
12/11 13:49:39.363773 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.363795 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC44E3F0 synchronization task has been failed!
12/11 13:49:39.367604 a90 Common: *** iscsi_tcp_dispatch: request execution timed out (15 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144716, maxcmdsn = 8144963.
12/11 13:49:39.367659 a90 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144716, itt = 8150112)
12/11 13:49:39.367689 a90 Common: *** iSCSIAsyncCommandCallback: User request 000002C7E9553068 has failed due to unidentified error.
12/11 13:49:39.367711 a90 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.367726 a90 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29CC20 synchronization task has been failed!
12/11 13:49:39.379845 5848 Common: *** iscsi_tcp_dispatch: request execution timed out (16 ms) for target iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03(events[0] = 0x300, events[1] = 0x300), cmdsn = 8144718, expcmdsn = 8144717, maxcmdsn = 8144964.
12/11 13:49:39.379906 5848 HA: iscsi_timeout_scan: PDU is timeouted in WAITQUEUE(opcode = 0x1, cmdSn = 8144717, itt = 8150113)
12/11 13:49:39.379930 5848 Common: *** iSCSIAsyncCommandCallback: User request 000002C7EAA529C8 has failed due to unidentified error.
12/11 13:49:39.379951 5848 HA: *** CReadWriteSyncTask::partner_request_completed: Remote write(async) failed, SystemError = 0x507, ScsiStatus = 0x0, SenseCode = 0x0, AdSenseCode = 0x0, AdSenseQualifier = 0x0
12/11 13:49:39.379982 5848 HA: CReadWriteSyncTask::complete_task: The 0x000002C7EC29BAE0 synchronization task has been failed!
12/11 13:49:44.076022 26ec func: >>> HANode::onScsiOp_CustomControl_HeartbeatPing
12/11 13:49:44.076199 26ec Common: CStarWindStorageDevice::getLastPerfDegEventElapsedTimeMs: perfDegEventElapsedTime = -1
12/11 13:49:44.076223 26ec func: <<< HANode::onScsiOp_CustomControl_HeartbeatPing
12/11 13:49:44.077422 26ec func: >>> CHAPartnerNode::ShutdownNode
12/11 13:49:44.077453 26ec func: >>> CHAPartnerNode::SetSyncStatus
12/11 13:49:44.077465 26ec HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/11 13:49:44.077479 26ec func: <<< CHAPartnerNode::SetSyncStatus
12/11 13:49:44.077490 26ec func: <<< CHAPartnerNode::ShutdownNode
12/11 13:49:44.079465 32b8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
12/11 13:49:44.087965 5378 func: >>> HANode::reportEvent
12/11 13:49:44.088036 5378 func: >>> iScsiServer::sendNotification
12/11 13:49:44.088064 5378 func: >>> CEventDataBase::AddRecord
12/11 13:49:44.088078 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
12/11 13:49:44.088196 5378 func: <<< CEventDataBase::AddRecord
12/11 13:49:44.088210 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
12/11 13:49:44.089501 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
12/11 13:49:44.089536 5378 func: <<< iScsiServer::sendNotification
12/11 13:49:44.089551 5378 func: <<< HANode::reportEvent
12/11 13:49:44.089562 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
12/11 13:49:47.298545 26ec func: >>> CHAPartnerNode::ShutdownNode
12/11 13:49:47.298613 26ec func: >>> CHAPartnerNode::SetSyncStatus
12/11 13:49:47.298626 26ec HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/11 13:49:47.298640 26ec func: <<< CHAPartnerNode::SetSyncStatus
12/11 13:49:47.298652 26ec func: <<< CHAPartnerNode::ShutdownNode
12/11 13:49:47.303065 28f0 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
12/11 13:49:47.304722 28f0 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D2550FEF58 (value is 0x0), p_puliEndSector = 0x000000D2550FEF50 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
12/11 13:49:47.304804 28f0 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
12/11 13:49:47.304836 28f0 func: >>> CHAPartnerNode::SetSyncStatus
12/11 13:49:47.304848 28f0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/11 13:49:47.304862 28f0 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
12/11 13:49:47.304875 28f0 func: >>> HANode::reportEvent
12/11 13:49:47.304890 28f0 func: >>> iScsiServer::sendNotification
12/11 13:49:47.304902 28f0 func: >>> CEventDataBase::AddRecord
12/11 13:49:47.304913 28f0 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
12/11 13:49:47.304986 28f0 func: <<< CEventDataBase::AddRecord
12/11 13:49:47.304998 28f0 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
12/11 13:49:47.305757 28f0 Srv: iScsiServer::sendNotification: Send reaction finish.
12/11 13:49:47.305788 28f0 func: <<< iScsiServer::sendNotification
12/11 13:49:47.305801 28f0 func: <<< HANode::reportEvent
12/11 13:49:47.305813 28f0 func: <<< CHAPartnerNode::SetSyncStatus
12/11 13:49:47.306141 6058 func: >>> CHAPartnerNode::SetSyncStatus
12/11 13:49:47.306185 6058 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/11 13:49:47.306210 6058 func: <<< CHAPartnerNode::SetSyncStatus
12/11 13:49:47.310024 5378 func: >>> HANode::reportEvent
12/11 13:49:47.310072 5378 func: >>> iScsiServer::sendNotification
12/11 13:49:47.310088 5378 func: >>> CEventDataBase::AddRecord
12/11 13:49:47.310101 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
12/11 13:49:47.310180 5378 func: <<< CEventDataBase::AddRecord
12/11 13:49:47.310193 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
12/11 13:49:47.310998 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
12/11 13:49:47.311033 5378 func: <<< iScsiServer::sendNotification
12/11 13:49:47.311047 5378 func: <<< HANode::reportEvent
12/11 13:49:47.311060 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Fri Dec 12, 2025 6:05 am
by yaroslav (staff)
Thise are network timeouts. If enabled, disable jumbo packets.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Fri Dec 12, 2025 12:02 pm
by Electrum
Hello,

I assume you mean setting both back to an MTU of 1500? Or also disabling jump frames within the NICs?
For clarity, here is a simplified topology diagram (https://imgur.com/a/524M0s0)

Sync and iSCSI are two distinct vSwitches both utilized for sync.

vSwitch_01 and vSwitch_02 are utilized for heartbeats (as well as VM traffic) and go to two distinct switches. I did just run NTTTCP (attached) and I noticed errors. The 25G links are DAC cables, I don't have 25G DACs handy, but I do have 10G. I'm wondering if you think it's worth trying with those and rerunning the test? If NTTTCP doesn't show errors, I can run a resync.

NTTTCP https://imgur.com/a/U3xYHc8

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Fri Dec 12, 2025 12:39 pm
by yaroslav (staff)
Set MTU to 1500 on the entire network stack, please (i.e., vSwitches, VMKs, inside the VMs)
Make sure you don't team replicatio liks or use any other form of network abstraction.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Fri Dec 12, 2025 9:15 pm
by Electrum
Hello,

I've done that. I even reinstalled windows server 2022 on the second host. I even designated 1 of the 25G nics for iSCSI only, 1 for sync only. Here are the latest logs:

Code: Select all

12/12 15:54:43.151648 23ac HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.151668 23ac func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.151688 23ac func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.151699 23ac HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.151710 23ac func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.151721 23ac func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.154027 609c Common: *** sw_common::Sw_Disk_Header::backup: Asynchronous header 'C:\Program Files\StarWind Software\StarWind\headers\datastore02\datastore02_HA.swdsk' operation in progress!
12/12 15:54:43.154888 2614 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7F0552310) partnerRequest(0x000002C7E86A4A30) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(4096), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 9703, ms DiffTimeCompleteINIT = 9703 ms, DiffTimeCompleteEXEC = 9703 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.155278 2cd8 HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
12/12 15:54:43.155638 524c HA: CHAPartnerNode::AfterSynchronization: Event: Full synchronization has been completed (error: 1237)
12/12 15:54:43.159075 5378 func: >>> HANode::reportEvent
12/12 15:54:43.159103 5378 func: >>> iScsiServer::sendNotification
12/12 15:54:43.159109 5378 func: >>> CEventDataBase::AddRecord
12/12 15:54:43.159114 5378 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
12/12 15:54:43.159150 5378 func: <<< CEventDataBase::AddRecord
12/12 15:54:43.159155 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
12/12 15:54:43.159741 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
12/12 15:54:43.159759 5378 func: <<< iScsiServer::sendNotification
12/12 15:54:43.159765 5378 func: <<< HANode::reportEvent
12/12 15:54:43.159770 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'
12/12 15:54:43.161288 23ac func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.161821 23ac func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.161870 23ac HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7F0311040) partnerRequest(0x000002C7EA4DBB50) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 14515 ms, DiffTimeCompleteEXEC = 14515 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.161913 16b0 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.162293 16b0 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.162321 16b0 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.162334 16b0 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.162346 16b0 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.162352 16b0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.162360 16b0 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.162366 16b0 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.167279 2698 func: >>> HANode::reportEvent
12/12 15:54:43.167309 2698 func: >>> iScsiServer::sendNotification
12/12 15:54:43.167317 2698 func: >>> CEventDataBase::AddRecord
12/12 15:54:43.167322 2698 EventDB: CEventDataBase::AddRecord: Code 783, severity 1, additional strings: 2
12/12 15:54:43.167356 2698 func: <<< CEventDataBase::AddRecord
12/12 15:54:43.167360 2698 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 783).
12/12 15:54:43.167855 2698 Srv: iScsiServer::sendNotification: Send reaction finish.
12/12 15:54:43.167872 2698 func: <<< iScsiServer::sendNotification
12/12 15:54:43.167878 2698 func: <<< HANode::reportEvent
12/12 15:54:43.167882 2698 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_COMPLETE_WITH_ERROR, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'
12/12 15:54:43.176182 16b0 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.176797 16b0 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.176822 3840 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.176835 16b0 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7F054BBF0) partnerRequest(0x000002C7F3CCC5F0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(16384), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 13406 ms, DiffTimeCompleteEXEC = 13406 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.177634 3840 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.177667 3840 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.177682 3840 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.177697 3840 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.177705 3840 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.177714 3840 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.177723 3840 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.181568 3840 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.182207 3840 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.182221 6b48 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.182238 3840 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EA474FB0) partnerRequest(0x000002C7EFED8410) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 12125 ms, DiffTimeCompleteEXEC = 12125 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.182818 6b48 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.182847 6b48 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.182857 6b48 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.182868 6b48 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.182873 6b48 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.182879 6b48 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.182884 6b48 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.186195 6b48 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.186575 6b48 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.186580 63b8 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.186607 6b48 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EF4EC100) partnerRequest(0x000002C7EF48C840) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 11344 ms, DiffTimeCompleteEXEC = 11344 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.186983 63b8 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.187031 63b8 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.187056 63b8 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.187070 63b8 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.187078 63b8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.187093 63b8 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.187104 63b8 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.191608 63b8 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.192151 63b8 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.192156 5d4c HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.192179 63b8 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EA472670) partnerRequest(0x000002C7E94CDF50) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10563 ms, DiffTimeCompleteEXEC = 10563 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.192525 5d4c func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.193629 5d4c HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.193656 5d4c func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.193677 5d4c func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.193688 5d4c HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.193700 5d4c func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.193711 5d4c func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.196647 5d4c func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.197259 5d4c func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.197272 13e0 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.197310 5d4c HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EA92A770) partnerRequest(0x000002C7F35854B0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10453 ms, DiffTimeCompleteEXEC = 10453 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.197631 13e0 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.197661 13e0 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.197679 13e0 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.197699 13e0 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.197709 13e0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.197721 13e0 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.197731 13e0 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.201021 13e0 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.201516 13e0 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.201535 29a4 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.201542 13e0 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EF4ED910) partnerRequest(0x000002C7EF48BFA0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10468 ms, DiffTimeCompleteEXEC = 10468 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.201899 29a4 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.201942 29a4 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.201958 29a4 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.201974 29a4 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.201983 29a4 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.201993 29a4 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.202002 29a4 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.206876 29a4 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.207546 29a4 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.207553 5070 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.207595 29a4 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EA543090) partnerRequest(0x000002C7EAAC12C0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(4096), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10468 ms, DiffTimeCompleteEXEC = 10468 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.208218 5070 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.208240 5070 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.208248 5070 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.208257 5070 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.208261 5070 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.208266 5070 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.208270 5070 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.211223 5070 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.215420 5070 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.215430 1408 HA: HASyncNode::solvePartnerCmdSendFailure: Remote send failure global lock
12/12 15:54:43.215451 5070 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EA471C20) partnerRequest(0x000002C7F46A4970) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(8192), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10484 ms, DiffTimeCompleteEXEC = 10484 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:43.216006 1408 func: >>> HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.216035 1408 HA: HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat: We don't need to solve partner send failure because partner node isn't synchronized (sync status = 3).
12/12 15:54:43.216048 1408 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:43.216061 1408 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:43.216068 1408 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'.
12/12 15:54:43.216075 1408 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:43.216082 1408 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:43.220527 1408 func: <<< HASyncNode::solvePartnerCmdSendFailureUsingHeartbeat
12/12 15:54:43.221047 1408 func: <<< HASyncNode::solvePartnerCmdSendFailure
12/12 15:54:43.221077 1408 HA: Ssc_Request_Task::complete: Warning(Partner send): sscRequestTask(0x000002C7EF4E7560) partnerRequest(0x000002C7E9AB4AB0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0) buf_size(4096), PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 10000, ms DiffTimeCompleteINIT = 10344 ms, DiffTimeCompleteEXEC = 10344 ms. Current target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h01-datastore02', Partner target: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore02'!
12/12 15:54:44.022994 1c20 func: >>> CHAPartnerNode::ShutdownNode
12/12 15:54:44.023080 1c20 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:44.023094 1c20 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 3, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/12 15:54:44.023108 1c20 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:44.023165 1c20 func: <<< CHAPartnerNode::ShutdownNode
12/12 15:54:44.027881 30a8 HA: HANode::onScsiOp_CustomControl_SynchronizationRequest: The partner iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03 has requested the 1 synchronization type.
12/12 15:54:44.029433 30a8 HA: CHAPartnerNode::Synchronize: ENTERed with params: p_SyncType = 1, p_ulSnapshotID = 0(0x0), p_pStorageDevice = 0x000002C7ECEDC4B0, p_pSyncBarrier = 0x000002C7ECEDC858, p_puliStartSector = 0x000000D253BFF4F8 (value is 0x0), p_puliEndSector = 0x000000D253BFF4F0 (value is 0xAEFFFFF), p_ulIOBlockSizeInSectors = 64, p_ulSectorSizeInUCHARs = 4096, p_bContinue = no, p_bVaaiWriteSameSupport = yes
12/12 15:54:44.029501 30a8 HA: HANode::updatePartnerStatus: Statuses is already actual, we don't need to update them.
12/12 15:54:44.029535 30a8 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:44.029547 30a8 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 3) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/12 15:54:44.029560 30a8 HA: CHAPartnerNode::SetSyncStatus: Event: Partner device changed sync status to 2 from 3. (partner: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03')
12/12 15:54:44.029573 30a8 func: >>> HANode::reportEvent
12/12 15:54:44.029587 30a8 func: >>> iScsiServer::sendNotification
12/12 15:54:44.029601 30a8 func: >>> CEventDataBase::AddRecord
12/12 15:54:44.029612 30a8 EventDB: CEventDataBase::AddRecord: Code 770, severity 2, additional strings: 2
12/12 15:54:44.029688 30a8 func: <<< CEventDataBase::AddRecord
12/12 15:54:44.029700 30a8 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 770).
12/12 15:54:44.030494 30a8 Srv: iScsiServer::sendNotification: Send reaction finish.
12/12 15:54:44.030527 30a8 func: <<< iScsiServer::sendNotification
12/12 15:54:44.030541 30a8 func: <<< HANode::reportEvent
12/12 15:54:44.030553 30a8 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:44.031112 bc0 func: >>> CHAPartnerNode::SetSyncStatus
12/12 15:54:44.031161 bc0 HA: CHAPartnerNode::SetSyncStatus: Params(new status = 2, old status = 2) ENTERed for partner 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03'.
12/12 15:54:44.031180 bc0 func: <<< CHAPartnerNode::SetSyncStatus
12/12 15:54:44.036093 5378 func: >>> HANode::reportEvent
12/12 15:54:44.036145 5378 func: >>> iScsiServer::sendNotification
12/12 15:54:44.036161 5378 func: >>> CEventDataBase::AddRecord
12/12 15:54:44.036175 5378 EventDB: CEventDataBase::AddRecord: Code 780, severity 1, additional strings: 3
12/12 15:54:44.036250 5378 func: <<< CEventDataBase::AddRecord
12/12 15:54:44.036263 5378 Srv: iScsiServer::sendNotification: Send reaction start (reaction type 2, msg code 780).
12/12 15:54:44.037224 5378 Srv: iScsiServer::sendNotification: Send reaction finish.
12/12 15:54:44.037263 5378 func: <<< iScsiServer::sendNotification
12/12 15:54:44.037278 5378 func: <<< HANode::reportEvent
12/12 15:54:44.037291 5378 HA: HADeviceEventTask::execute: Event: HA_NN_PARTNER_NODE_SYNCHRONIZATION_STARTED, Partner node: 'iqn.2008-08.com.starwindsoftware:home-hypv-h02-datastore03', Synchronization type: 'Full'

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Sat Dec 13, 2025 11:29 am
by yaroslav (staff)
Please recreate replica. Make sure to purge the /headers and the corresponding files from /mnt/mountpointname.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Sat Dec 13, 2025 3:14 pm
by Electrum
Hello,

Do you mean delete the .img files on Host02 and the applicable headers? If so, I did that when I reimaged Host02 and established a fresh sync.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Sat Dec 13, 2025 8:12 pm
by yaroslav (staff)
Thanks for your update. Please share the full log with me.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Sun Dec 14, 2025 3:47 pm
by Electrum
I can't post the full log, the forums don't seem to allow me to. But here are the logs for the last day from Host01. https://pastebin.com/CPTBAxk4

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Sun Dec 14, 2025 10:38 pm
by yaroslav (staff)
Hi, log is full of network timeouts. Please connect two servers with the direct cable if possible.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Mon Dec 15, 2025 12:33 pm
by Electrum
Hello,

These servers are directly connected via 25G NICs. 1 is used for iSCSI, the other is used for Sync.

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Mon Dec 15, 2025 1:01 pm
by yaroslav (staff)
Thanks for your update.
Can you please tell me more about the system? If it is a VMware ESXi-based system, tell me more about the VM spec that you are using to run StarWind VSAN?

Re: Hyper-V 2 node cluster consistently restarts full sync

Posted: Mon Dec 15, 2025 1:40 pm
by Electrum
Hello,

These are Hyper-V hosts, running Windows Server 2022 Datacenter. Starwind vSAN is running as a windows service on them.
All times are UTC
Page 1 of 3

Powered by phpBB® Forum Software © phpBB Limited