Unable to add new targets

[Michael.Blanchard2]

I have had an existing startwind deployment for the past 11 months, and it's been great, now for some reason i'm unable to add any new devices to synchronization, but existing devices continue to sync with no issues.


Adding first device works with no issue, But when I go to the synchronization manager to add the other server, it runs through, and says that all the interfaces are down, and the syncrhronization never starts. It does the same thing no matter what storage i put the device on. In the logs I see authentication failures, but i'm not using any authentication settings, and I see stuff like this in log:
7 16:34:46.438 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.438 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: EXITing with failure, SendCustomControlScsiCommand(HA_CHANNEL_TYPE_HEARTBEAT) failed, error code 1168, scsi status = 0!
9/17 16:34:46.438 1368 HA: *** CHAPartnerNode::SendGetPartnerNodeInfoCommandV6: EXITing with failure, partner node version update failed!
9/17 16:34:46.454 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: SendCustomControlScsiCommand(HA_CHANNEL_TYPE_SYNC) failed, error code 1168, scsi status = 0!
9/17 16:34:46.454 1368 HA: CHAPartnerNode::SendPartnerNodeVersionRequestCommand: Try to get partner node version through heartbeat channel.
9/17 16:34:46.454 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: EXITing with failure, SendCustomControlScsiCommand(HA_CHANNEL_TYPE_HEARTBEAT) failed, error code 1168, scsi status = 0!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNetControlInterfacesRequestCommand: EXITing with failure, partner node version 0x0 is not supported or invalid. Nothing will be sent!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::GetNetControlInterfacesList: EXITing with failure, SendPartnerNetInterfacesRequestCommand() failed, error code!
9/17 16:34:46.454 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: SendCustomControlScsiCommand(HA_CHANNEL_TYPE_SYNC) failed, error code 1168, scsi status = 0!
9/17 16:34:46.454 1368 HA: CHAPartnerNode::SendPartnerNodeVersionRequestCommand: Try to get partner node version through heartbeat channel.
9/17 16:34:46.454 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: EXITing with failure, SendCustomControlScsiCommand(HA_CHANNEL_TYPE_HEARTBEAT) failed, error code 1168, scsi status = 0!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendGetPartnerNodeInfoCommandV6: EXITing with failure, partner node version update failed!
9/17 16:34:46.454 1368 HA: *** CHAPartnerISCSIChannelManager::SendCustomControlScsiCommand: Valid channel not found!
9/17 16:34:46.454 1368 HA: *** CHAPartnerNode::SendPartnerNodeVersionRequestCommand: SendCustomControlScsiCommand(HA_CHANNEL_TYPE_SYNC) failed, error code 1168, scsi status = 0!
9/17 16:34:46.454 1368 HA: CH

I've tried updating software, and removing and reloading software and I always get same errors and results.

[Boris (staff)]

What is your build?
How do you try adding new devices?

[Michael.Blanchard2]

Both are build 12393.

Add device (advanced) and use the same settings as the others, no cache, 512 cluster size, then I select the same network interfaces as the others. I then right click the device, replication manager, and no matter if I add the other node by IP, fqdn or shortname, it creates the replication, then sits, says it cannot replicate, and it says all the networks are down. What's really weird is I can connect to the device with the local iscsi initiator locally on both servers, but as soon as I enable replication on server b, I can no longer connect locally with the iscsi initiator, but will connect remotely. But the other server, server a, will actually work locally, but it will not map the remote one.ut no matter where server the device is created on and replication ran, it does the same thing when it doesn't sync.

Server a process:
1. New target and device created on server a
2. replication manager, enable replication to server b.
3. replication setup finishes, says all network interfaces are down
4. Look on server a, the target I originally made is there, but the device is now disconnected from the target and is orphaned, but server b has a target and device, and is saying it's saying synchronization is broken. going back to server a and reconnecting it doesn't help, if I redo it, it does the same thing again.

Server b: process:
1. Create new device and target on server b.
2. replication manager, enable replication to server a
3. replication finishes, says all network interfaces are down
4. look on server b, and the target / device are there, but it says un synchronized. I go to server a, and it has the new target, but the new device is unconnected to that same target.

during all this, I reconnected all the other existing targets and devices to server b with no issue and they are resynching now, the issue only happens with any new storage, no matter which server the new target is created and replication is enabled from.

[Michael.Blanchard2]

ok, this is fixed. I went through and re-installed the latest starwind on both of the nodes and things are once again working. I'm thinking there was an upgrade issue, or something broke on some of the software. I uninstalled it, rebooted it, deleted the starwind directory, and then re-installed it. I was then able to re-establish a connection with existing drives, and also start a new target / device and have it sync correctly

[Boris (staff)]

Great to know you have resolved this.