200 Failed: can't find partner node. Hyper-v HyperConverged!

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
madmongoose
Posts: 16
Joined: Mon Aug 21, 2017 8:05 am

Mon Aug 21, 2017 8:25 am

Hello, Starwind Team!

Sorry for my English.

We actively use Starwind in our company. But there was a problem with synchronization, after a long disconnection of one node.



There are two nodes:

#Import-Module StarWindX

try
{
$server = New-SWServer -host 10.131.1.61 -port 3261 -user svc -password *

$server.Connect()

$firstNode = new-Object Node

$firstNode.ImagePath = "My computer\d\ais-vsan\ais-iscsi-fast"
$firstNode.ImageName = "ais-iscsi-fast"
$firstNode.Size = 2048000
$firstNode.CreateImage = $true
$firstNode.TargetAlias = "ais-iscsi-fast"
$firstNode.AutoSynch = $true
$firstNode.SyncInterface = "#p2=172.31.1.62:3260"
$firstNode.HBInterface = "#p2=10.131.1.62:3260"
$firstNode.CacheSize = 8192
$firstNode.CacheMode = "wb"
$firstNode.PoolName = ""
$firstNode.SyncSessionCount = 1
$firstNode.ALUAOptimized = $true

#
# device sector size. Possible values: 512 or 4096(May be incompatible with some clients!) bytes.
#
$firstNode.SectorSize = 512

#
# 'SerialID' should be between 16 and 31 symbols. If it not specified StarWind Service will generate it.
# Note: Second node always has the same serial ID. You do not need to specify it for second node
#
#firstNode.SerialID = "050176c0b535403ba3ce06286e33eab"

$secondNode = new-Object Node

$secondNode.HostName = "10.131.1.62"
$secondNode.HostPort = "3261"
$secondNode.Login = "svc"
$secondNode.Password = "*"
$secondNode.ImagePath = "My computer\d\ais-vsan\ais-iscsi-fast"
$secondNode.ImageName = "ais-iscsi-fast"
$secondNode.Size = 2048000
$secondNode.CreateImage = $true
$secondNode.TargetAlias = "ais-iscsi-fast"
$secondNode.AutoSynch = $true
$secondNode.SyncInterface = "#p1=172.31.1.61:3260"
$secondNode.HBInterface = "#p1=10.131.1.61:3260"
$secondNode.ALUAOptimized = $true

$device = Add-HADevice -server $server -firstNode $firstNode -secondNode $secondNode -initMethod "Clear"

$syncState = $device.GetPropertyValue("ha_synch_status")

while ($syncState -ne "1")
{
#
# Refresh device info
#
$device.Refresh()

$syncState = $device.GetPropertyValue("ha_synch_status")
$syncPercent = $device.GetPropertyValue("ha_synch_percent")

Start-Sleep -m 2000

Write-Host "Synchronizing: $($syncPercent)%" -foreground yellow
}
}
catch
{
Write-Host "Exception $($_.Exception.Message)" -foreground red
}

$server.Disconnect()

Everything worked fine, but with long disconnection of the second node, the synchronization ends, or does not start at all.

An attempt is made to synchronize an error "Device not synchronized. Synchronize current node from partner 'iqn.2008-08.com.starwindsoftware:10.131.1.61-ais-iscsi-fast'
Exception Error:
200 Failed: can't find partner node."

When trying to solve the problem, it was found that the resources of the cluster (ip address 10.131.1.34, 10.131.1.70) listen for the ports of Starwind 3260.
2.png
2.png (37.36 KiB) Viewed 20665 times
Previously, while the Management Console was available, we could see in it these interfaces (ip address 10.131.1.34, 10.131.1.70), which Starwind automatically adds. And we could remove them from there and the synchronization passed normally. How should we be in the case of the Free version?

I am hope for your help.
Last edited by madmongoose on Mon Aug 28, 2017 8:55 am, edited 2 times in total.
Sergey (staff)
Staff
Posts: 86
Joined: Mon Jul 17, 2017 4:12 pm

Mon Aug 21, 2017 2:58 pm

Unfortunately, it is not possible to remove network interface with PowerShell now. We will add this functionality with the next update, within a month approximately.
madmongoose
Posts: 16
Joined: Mon Aug 21, 2017 8:05 am

Mon Aug 28, 2017 6:50 am

Hello!

I have created a whole new configuration from scratch. The initial synchronization ends perfectly. When testing the synchronization (hard reboot of one of the servers) an error occurs. And never once did not get to make Starwind work. Please help us to understand.

P.S.
I also want to note that Starwind listens for interfaces not specified in the configuration.

This is my config:
#Import-Module StarWindX

try
{
$server = New-SWServer -host 10.131.1.61 -port 3261 -user root -password starwind

$server.Connect()

$firstNode = new-Object Node

$firstNode.ImagePath = "My computer\d\ais-vsan\ais-iscsi-fs"
$firstNode.ImageName = "ais-iscsi-fs"
$firstNode.Size = 1048576
$firstNode.CreateImage = $true
$firstNode.TargetAlias = "ais-iscsi-fs"
$firstNode.AutoSynch = $true
$firstNode.SyncInterface = "#p2=172.31.1.62:3260"
$firstNode.HBInterface = "#p2=172.31.2.62:3260"
$firstNode.CacheSize = 4096
$firstNode.CacheMode = "wb"
$firstNode.PoolName = ""
$firstNode.SyncSessionCount = 1
$firstNode.ALUAOptimized = $true

#
# device sector size. Possible values: 512 or 4096(May be incompatible with some clients!) bytes.
#
$firstNode.SectorSize = 512

#
# 'SerialID' should be between 16 and 31 symbols. If it not specified StarWind Service will generate it.
# Note: Second node always has the same serial ID. You do not need to specify it for second node
#
#$firstNode.SerialID = "050176c0b535403ba3ce02102e33eab"

$secondNode = new-Object Node

$secondNode.HostName = "10.131.1.62"
$secondNode.HostPort = "3261"
$secondNode.Login = "root"
$secondNode.Password = "starwind"
$secondNode.ImagePath = "My computer\d\ais-vsan\ais-iscsi-fs"
$secondNode.ImageName = "ais-iscsi-fs"
$secondNode.Size = 1048576
$secondNode.CreateImage = $true
$secondNode.TargetAlias = "ais-iscsi-fs"
$secondNode.AutoSynch = $true
$secondNode.SyncInterface = "#p1=172.31.1.61:3260"
$secondNode.HBInterface = "#p1=172.31.2.61:3260"
$secondNode.ALUAOptimized = $true

$device = Add-HADevice -server $server -firstNode $firstNode -secondNode $secondNode -initMethod "Clear"

$syncState = $device.GetPropertyValue("ha_synch_status")

while ($syncState -ne "1")
{
#
# Refresh device info
#
$device.Refresh()

$syncState = $device.GetPropertyValue("ha_synch_status")
$syncPercent = $device.GetPropertyValue("ha_synch_percent")

Start-Sleep -m 2000

Write-Host "Synchronizing: $($syncPercent)%" -foreground yellow
}
}
catch
{
Write-Host "Exception $($_.Exception.Message)" -foreground red
}

$server.Disconnect()

This is error: Exception Error: 200 Failed: can't find partner node..
error
error
error.jpg (211.45 KiB) Viewed 20617 times
cluster-address
cluster-address
address.jpg (56.56 KiB) Viewed 20617 times
ports
ports
ports.jpg (107.2 KiB) Viewed 20617 times
madmongoose
Posts: 16
Joined: Mon Aug 21, 2017 8:05 am

Tue Aug 29, 2017 7:11 am

Dear Starwind team!

Please help in my problem. We can not work normally.
error2
error2
error2.jpg (143.85 KiB) Viewed 20598 times
In the log, a cyclic error:

8/29 10:07:25.076 214c Tgt: iqn.2008-08.com.starwindsoftware:10.131.1.62-ais-iscsi-fs: can't register session. The device 'HAImage2' is not ready.
8/29 10:07:25.076 214c T[5041,1]: *ERROR* Login request: device open failed.
8/29 10:07:25.076 484 C[5041], IN_LOGIN: Event - LOGIN_REJECT.
8/29 10:07:25.076 484 C[5041], IN_LOGIN: T7.
8/29 10:07:25.076 214c Srv: WSARecv() failed with error 10054 (0x2746)!
8/29 10:07:25.076 214c C[5041], IN_LOGIN: recvData returned 10054 (0x2746)!
8/29 10:07:25.076 1814 S[5041]: ~Session
8/29 10:07:27.107 7cc Srv: Accepted iSCSI connection from 10.131.1.61:58516 to 10.131.1.62:3260. (Id = 0x5042)
8/29 10:07:27.107 7cc S[5042]: Session (000001805B354400)
8/29 10:07:27.107 7cc C[5042], FREE: Event - CONNECTED.
8/29 10:07:27.107 7cc C[5042], XPT_UP: T3.
8/29 10:07:27.107 1c8 C[5042], XPT_UP: Login request: ISID 0x400001370000, TSIH 0x0000.
8/29 10:07:27.107 1c8 C[5042], XPT_UP: Event - LOGIN.
8/29 10:07:27.107 1c8 C[5042], IN_LOGIN: T4.
8/29 10:07:27.107 1c8 Params: <<< String param 'InitiatorName': received 'iqn.1991-05.com.microsoft:ais-srv-hv-01.ai-sys.ru', accepted 'iqn.1991-05.com.microsoft:ais-srv-hv-01.ai-sys.ru'
8/29 10:07:27.107 1c8 Params: <<< Enum param 'SessionType': received 'Normal', accepted 'Normal'
8/29 10:07:27.107 1c8 Params: <<< String param 'TargetName': received 'iqn.2008-08.com.starwindsoftware:10.131.1.62-ais-iscsi-fs', accepted 'iqn.2008-08.com.starwindsoftware:10.131.1.62-ais-iscsi-fs'
8/29 10:07:27.107 1c8 Params: <<< Enum param 'AuthMethod': received 'None', accepted 'None'
8/29 10:07:27.107 1c8 HA: CHADevice::register_session: Client initiator iqn.1991-05.com.microsoft:ais-srv-hv-01.ai-sys.ru is trying to register a session within the 'iqn.2008-08.com.starwindsoftware:10.131.1.62-ais-iscsi-fs' target... (sessId = 0x5042, initiatorNameIsid = iqn.1991-05.com.microsoft:ais-srv-hv-01.ai-sys.ru,400001370000)
8/29 10:07:27.107 1c8 HA: CHADevice::register_session: Unable to register the new client session. The node is not active!
8/29 10:07:27.107 1c8 HA: CHADevice::register_session: Return code 21.
8/29 10:07:27.107 1c8 Tgt: iqn.2008-08.com.starwindsoftware:10.131.1.62-ais-iscsi-fs: can't register session. The device 'HAImage2' is not ready.
8/29 10:07:27.107 1c8 T[5042,1]: *ERROR* Login request: device open failed.
8/29 10:07:27.107 10f0 C[5042], IN_LOGIN: Event - LOGIN_REJECT.
8/29 10:07:27.107 10f0 C[5042], IN_LOGIN: T7.
8/29 10:07:27.107 1c8 Srv: WSARecv() failed with error 10054 (0x2746)!
8/29 10:07:27.107 1c8 C[5042], IN_LOGIN: recvData returned 10054 (0x2746)!
8/29 10:07:27.107 21ec S[5042]: ~Session

Help!
madmongoose
Posts: 16
Joined: Mon Aug 21, 2017 8:05 am

Tue Aug 29, 2017 8:37 am

again..
error3
error3
error3.jpg (173.96 KiB) Viewed 20592 times
Sergey (staff)
Staff
Posts: 86
Joined: Mon Jul 17, 2017 4:12 pm

Tue Aug 29, 2017 9:07 am

Hello, madmongoose. We need additional information regarding this issue. Could you please open a ticket here https://www.starwindsoftware.com/support-form with a link to this forum thread.
Also, please, collect logs from both nodes according to this guide https://knowledgebase.starwindsoftware. ... ogs-bat-2/ and send it to us.
madmongoose
Posts: 16
Joined: Mon Aug 21, 2017 8:05 am

Tue Aug 29, 2017 9:39 am

I sent two support requests. In one request, it failed to attach logs from both servers.

Thank you!
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Tue Sep 05, 2017 11:18 am

Excellent! Ping me if you'd need any assistance here (speedup the process etc) ;)
madmongoose wrote:I sent two support requests. In one request, it failed to attach logs from both servers.

Thank you!
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
Sergey (staff)
Staff
Posts: 86
Joined: Mon Jul 17, 2017 4:12 pm

Tue Sep 05, 2017 1:26 pm

We have checked the logs and there are delays with writes. The node that is not synchronized does not have time to write data that it receives from the node with "Synchronized" status. Something (not StarWindService) very actively uses the storage on the node, which is not synchronized so it gives delays on write operations.
madmongoose wrote: Previously, while the Management Console was available, we could see in it these interfaces (ip address 10.131.1.34, 10.131.1.70), which Starwind automatically adds. And we could remove them from there and the synchronization passed normally. How should we be in the case of the Free version?
An update for the community:
there is a method in StarWindX library called SetServerParameter allowing to remove interfaces.

Example:

import-module StarWindX
$ s = New-SWServer 127.0.0.1 3261 root starwind
$ s.Connect ()

#Set list of disabled network interfaces:
$ s.SetServerParameter ("nolisten", "192.168.61.1:3260,192.168.60.1:3260")
Post Reply