Speed trouble

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
ViatoriPatuit
Posts: 4
Joined: Thu Aug 01, 2019 4:56 pm

Sat Aug 03, 2019 2:09 am

Greetings,

I am sorry if this is the wrong place to post this, or if I am posting something that has been covered extensively. I just do not know how to search for this issue.

My syncing and HA devices are slow. Super slow. like 464kps slow. If I move a file using a simple shared volume I get 110 MBS speeds. I realize that I am not super skilled at networking, but 110MBS seems reasonable, 464kpbs seems like something is horribly wrong.

I hope someone could point me to some resources that would help me isolate the bottleneck - it seems like Starwind but i really do not have any idea.

Here is my setup:

I am using 1 gb network cards with cat 5e cables to a gigabit switch (I know, I need to put redundancy in).

The NICs are teamed and I am using virtual ethernet ports (I saw that on a youtube and it seemed like the producer wasnt having speed issues). Each virtual NIC has its own IP address.

I do not see any other network slowdowns, and the task manager in windows servers shows the bottleneck as the network. disk, cpu and ram are almost at zero.

I am also testing this on windows server 2012 r2. I don't know if that matters.

I set the items (NVQ, Send buffer, Receive buffer and Jumbo frames) to the max or disable where appropriate on each NIC.

I am sorry if this has been covered, either some hints on how to search or a link to something that would help would be great.

Thanks for any and all help.
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Sat Aug 03, 2019 2:30 am

Get rid of virtual switches and virtual NICs for the links used for StarWind (iSCSI and Sync). Put direct links between the servers, no physical switch in between.
After this is done, perform the same test and tell us and results.
ViatoriPatuit
Posts: 4
Joined: Thu Aug 01, 2019 4:56 pm

Sat Aug 03, 2019 2:38 pm

I tried this morning - no dice.

I connected the Heartbeat, Live migration and iSCSI network ports directly together with just a cat 5e cable. I could ping the machines from each other and I ran a simple script as follows:
Import-Module StarWindX

try
{
Enable-SWXLog

$server = New-SWServer -host 192.168.7.12 -port 3261 -user root -password starwind

$server.Connect()

$firstNode = new-Object Node

$firstNode.HostName = "192.168.7.12"
$firstNode.ImagePath = "My computer\D\starwind"
$firstNode.ImageName = "masterImg21"
$firstNode.Size = 1024000
$firstNode.CreateImage = $true
$firstNode.TargetAlias = "targetha21"
$firstNode.AutoSynch = $true
$firstNode.SyncInterface = "#p2=10.10.10.50:3260"
$firstNode.HBInterface = "#p2=192.168.8.50:3260"
$firstNode.PoolName = "pool1"
$firstNode.SyncSessionCount = 1
$firstNode.ALUAOptimized = $true

#
# device sector size. Possible values: 512 or 4096(May be incompatible with some clients!) bytes.
#
$firstNode.SectorSize = 512

$secondNode = new-Object Node

$secondNode.HostName = "192.168.7.10"
$secondNode.HostPort = "3261"
$secondNode.Login = "root"
$secondNode.Password = "starwind"
$secondNode.ImagePath = "My computer\D\starwind"
$secondNode.ImageName = "partnerImg22"
$secondNode.Size = 12
$secondNode.CreateImage = $true
$secondNode.TargetAlias = "partnerha22"
$secondNode.AutoSynch = $true
$secondNode.SyncInterface = "#p1=10.10.10.70:3260"
$secondNode.HBInterface = "#p1=192.168.8.70:3260"
$secondNode.SyncSessionCount = 1
$secondNode.ALUAOptimized = $true

$device = Add-HADevice -server $server -firstNode $firstNode -secondNode $secondNode -initMethod "Clear"

while ($device.SyncStatus -ne [SwHaSyncStatus]::SW_HA_SYNC_STATUS_SYNC)
{
$syncPercent = $device.GetPropertyValue("ha_synch_percent")
Write-Host "Synchronizing: $($syncPercent)%" -foreground yellow

Start-Sleep -m 2000

$device.Refresh()
}
}
catch
{
Write-Host $_ -foreground red
}
finally
{
$server.Disconnect()
}
I am seeing the same network speeds now as I did with three and using the switch (the max is 500 kbps). it was still looking at a 45 minute sync process. That seems awful long for a TB of data (that can copy over the same network with a switch in 2 minutes).
ViatoriPatuit
Posts: 4
Joined: Thu Aug 01, 2019 4:56 pm

Sat Aug 03, 2019 3:51 pm

One quick edit - I am using cat 6 cables for the direct connection.

I did get the 1 TB sync time down to about 50 minutes, but that was on two nodes directly connected. That still seems like an awful long time - is it?
ViatoriPatuit
Posts: 4
Joined: Thu Aug 01, 2019 4:56 pm

Sat Aug 03, 2019 5:57 pm

One more edit -

I tried every combination of direct patching and switched networking. Nothing changed the speed of syncing between the servers. two or three servers were only different by about 20 minutes for 1TB.

I am begining to think that syncing is just a long process? How long should it take to sync 1 TB over 1 GBe cards?
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Wed Aug 07, 2019 1:12 am

Try testing your network throughput using iperf. That would show you what your networking hardware is capable of, and based on that you will be able to continue troubleshooting the bottleneck.
Post Reply