Network performance sporadically slow

[m.witte]

Hello dear forum,

I have the following problem with my vSAN FREE Storage.
Sporadically, on the system where I have connected the iSCSI storage, I get my sent data back as a response from the storage.

In the task manager or resource monitor I see for example:
Sent: 2.3GBit
Received: 2,3GBit

At this moment my performance is significantly slower than during the previous copy operations.

When this error does not occur the utilization of my network card looks like this for example:

Sent: 2.3GBit
Received: 2MBit

What is the problem ?

My configuration:
2x Lenovo x3650 M5 servers
4x Enterptise SSDs in RAID 5 per server
4x 10GBit Network Card


##My Script##

#common
$initMethod="Clear",
$size=10240,
$sectorSize=4096,
$failover=0,
#primary node
$imagePath="My Computer\D\TEST",
$imageName="TEST",
$createImage=$true,
$storageName="",
$targetAlias="TEST",
$autoSynch=$true,
$poolName="pool1",
$syncSessionCount=1,
$aluaOptimized=$true,
$cacheMode="wb",
$cacheSize="128",
$syncInterface="#p2=192.168.1.2:3260" -f $addr2,
$hbInterface="#p2=192.168.0.2:3260" -f $addr2,
$createTarget=$true,
#secondary node
$imagePath2="My Computer\D\TEST",
$imageName2="TEST",
$createImage2=$true,
$storageName2="",
$targetAlias2="TEST",
$autoSynch2=$true,
$poolName2="pool1",
$syncSessionCount2=1,
$aluaOptimized2=$false,
$cacheMode2=$cacheMode,
$cacheSize2=$cacheSize,
$syncInterface2="#p1=192.168.1.1:3260" -f $addr,
$hbInterface2="#p1=192.168.0.1:3260" -f $addr,
$createTarget2=$true
)

[yaroslav (staff)]

Welcome to StarWind Forum.

I get my sent data back as a response from the storage.

Could you kindly elaborate on this statement? It is not clear how the problem manifests itself.
Next time it happens, could you please check StarWind Management Console to see if everything is synchronized?

Also, I'd suggest adding a redundant heartbeat ("control") IP as described here https://forums.starwindsoftware.com/vie ... f=5&t=5743.
For SSDs, I'd also remove the Write-Back cache as described here https://knowledgebase.starwindsoftware. ... -l1-cache/. It may not improve performance for all-flash arrays, but makes the systems more complicated to manage due to possible full synchronization running https://knowledgebase.starwindsoftware. ... may-start/. See more about caching https://knowledgebase.starwindsoftware. ... rinciples/.

[m.witte]

Hello,

I have checked it again.
The system is sycronized.

Example scenario:

I copy a 5GB file.

In the task manager I see the load of my network card over which the storage is connected.

This looks, if everything is okay for example like this:
Sent: 1,5GBit
Received: 3 MBit

But when the problem occurs the load in the task manager looks like this:
Sent: 1,5GBit
Received: 1,5GBit

[yaroslav (staff)]

Are you copying over the management interface?
Do you have StarWind Management Console disconnected or does the storage go out of sync? If the latter is true, you are using management StarWind VSAN Synchronization which is not supported configuration as StarWind VSAN requires a dedicated channel for iSCSI and Synchronization. Please see more at https://www.starwindsoftware.com/system-requirements

[m.witte]

Hello,

I have set up the following networks:

1x 1GBit Management network
1x 1GBit Hearbeat network
1x 10GBit Sync network
1x 10GBit iSCSI Target network

The copy process takes place on the iSCSI network.
The management network etc. have no load during this time.

[yaroslav (staff)]

Excuse me but t is not clear what the problem is and how it manifests itself. High network utilization during copying is expected. As mentioned herein, iSCSI is to be used exclusively for iSCSI traffic (i.e., connecting the storage). It might be the case that you have storage connected wrong. Is it a hyperconverged system?
I think, the best it would be to trigger the issue, collect the logs, and share them here. Please collect the logs using StarWind Log Collector on all StarWind hosts as described here https://knowledgebase.starwindsoftware. ... collector/ and share them via Google disk, OneDrive, Sharepoint, etc., here.

[m.witte]

Thank you,

here is the link:
https://we.tl/t-Ctb7sovZ0n

[m.witte]

[yaroslav (staff)]

Will be back soon with log investigation.

[yaroslav (staff)]

Greetings,

Thank you for the logs and the diagram! I have reviewed the logs but I don't understand what exactly is the problem. StarWind HA devices seem to be in sync and go out of sync only during service restarts. I'd appreciate having the logs from the compute boxes to see if there are any "unexpected" target disconnects.

In the task manager I see the load of my network card over which the storage is connected.

This is related to i/o to your storage systems. If you are writing anything to the storage, it goes over iSCSI and replicates. Network traffic is expected here. May I ask how exactly are you copying?

I get my sent data back as a response from the storage.

This is also not clear.

P.s. You are using 2 different NICs here for sync and heartbeat. This is fine. Adding another heartbeat over iSCSI is still a good idea so you have some level of redundancy.