Starwind vSAN FREE HA slow sync when writing to the disk

milne301 · Thu Apr 29, 2021 1:52 pm

Hi Guys

I've been racking my brains with this for a while now so thought i would ask the experts here.

My Setup:

2 x 10G HP 5800 switches.
2 x DL380 G9's full of 300G 15K disks in RAID 50.
VMWare 6.7 installed on both.
Each server has 4 x 10G slots.
1 x management.
1 x vmotion.
2 x ISCSI setup as LACP both on switch and distributed switch. (MTU 9000)

I created 2 VM's on each host. Presented the RAW disks to these VM's making 6TB. Presented the distributed switch ports to the VM's for the vsync (4 x VMXNET3)

Installed starwind free vsan and setup 2 team'd 10Gbps making 2 x 20Gbps interfaces for syncing. Added an IP to each NIC (2 each side)

I set off the powershell script for the HA creation included the 2 NIC's either side for syncing. Cache is 128 wb.

Once this sync has been done i present this disk to vmware via ISCSI - adding the 4 IP's i have assign in order to get more paths to the disk. I then change this to Round robin and change the iops of the disk to 1 within VMware.

Once presented to VMware i create a windows VM on that disk and download a 10G file within. I then try and copy and paste the 10G file again testing write speeds.

few questions:

1) When i initiate the creation of the disk it uses 1 NIC on the primary and 2 NIC's on the secondary - why doesnt it use both nic's to sync when initially powering on the VM's?
2) I get good speeds when syncing for the first time powering the VM's on (9Gbps on both NIC's each way) - however when i try to write to the disk with ISCSI within the disk presented to VMware the speeds each way never go above 1.3-1.8Gbps each way. The speed of the writes go at about 3-600MB/s then slow to 1-200MB/s. I'd imagine this is because the NIC's are sending fast enough? or is this possibly disk saturation?

What i'm trying to acheive here is a 6TB HA DISK attached to VMware where i can create VM's on it and get as fast as a i can write speeds and sync speeds (not sure why these seem to cap at about 1 1.8Gbps when i have seen easy 9-10Gbps each NIC syncing initially.

Before you ask about SSD's i dont have the money to buy them or i would.

I think ive mentioned everything but may have missed some - any help is appreciated.

Thanks
Craig

Sat May 01, 2021 9:19 am

Welcome to StarWind Forum!

We do not recommend RAID 50 or 5 for HDDs, see the recommended RAID settings at https://knowledgebase.starwindsoftware. ... ssd-disks/.
We also do not recommend teaming. Please use dedicated connections for iSCSI and Sync. Is it possible to connect the servers directly? This guide may be helpful https://www.starwindsoftware.com/resour ... phere-6-5/. Just use individual vSwitches and iSCSI and Sync going over the separate network cards. How many physical cards do you have in your servers? Make sure to use 2 different network cards as described here https://www.starwindsoftware.com/system-requirements.
Please note that write-back cache may be the reason for full sync if StarWind VSAN service is hard-stopped (e.g., a PSOD of ESXi host, power supply failure, turning off VM, StarWind VSAN Service halt, etc.). Without cache, if that is done to one of your servers, you get fast sync.
Please also note that copying file is not the best benchmark as it may be copied via the Management link. See more about benchmarking at https://www.starwindsoftware.com/best-p ... practices/.

Let me answer your questions now.

1) When i initiate the creation of the disk it uses 1 NIC on the primary and 2 NIC's on the secondary - why doesnt it use both nic's to sync when initially powering on the VM's?

We recommend using separate NICs to ensure the best performance over iSCSI. When the same NIC is used for Sync, there is traffic mixing which affects the overall stability and performance

2) I get good speeds when syncing for the first time powering the VM's on (9Gbps on both NIC's each way) - however when i try to write to the disk with ISCSI within the disk presented to VMware the speeds each way never go above 1.3-1.8Gbps each way. The speed of the writes go at about 3-600MB/s then slow to 1-200MB/s. I'd imagine this is because the NIC's are sending fast enough? or is this possibly disk saturation?

Performance is not only about NICs but also storage. If your storage has slow writes/reads under 254 blocks, the overall performance will be bottlenecked by storage. Also, please note the virtualization layers I/O has to pass through.