Extremely slow sync data rate
Posted: Wed Jan 15, 2020 1:42 pm
I am just starting to tinker with setting up VSAN Free via powershell on my 2-node hyper-v server 2019 setup. I have created several "HA Devices" between my two nodes and each time have run into the same problem. Once I create the device, the "sync" uses only 1 interface at most, and seems to operate at a maximum of ~48-56kbs bandwidth.
[*] I have disabled the firewall rules for the "private" network profile and forced the networks to "private", as they come up as "public" by default in windows, not sure if there is something I should be doing differently when setting up direct ethernet links so that they are not "unidentified" and classified as "public" by the OS...
[/list]
I have enabled jumbo frames on all 7 adapters used in this test, and tested/verified them with
I set the jumbo frame size to 9014 using powershell:
Ping fails on forced fragmenting for any size greater than 8972, I suspect overhead in the packet header(s)? Not sure exactly.
Normal bandwidth seems OK; I have run iperf between the nodes over each connection and getting ~995 mbits/s with 0.001% loss with parameters like:
I initially tried multiple interfaces for both sync and iSCSI/HB on a 400gb image on a SATA drive. Later I tried a 1GB image with only 1 interface for sync, and 1 for iSCSI/HB. Same deal, ~48kbs sync rate. I also tried creating a second image (~2GB) while the 1GB was still syncing but using different interfaces. The traffic WAS split between the two interfaces, but the total bandwidth was the same! Each image sync rate dropped to half = ~24kbs! Once one completed, "full" 48kbs bandwidth was restored to the other image still syncing. Aditionally, there is an "initial" burst of ~2-4mbs once the device is initially created, probably the powershell talking to the "other" swserver to create the other image file I suspect.
Anyone got an idea of what I should look at next? Or if I there is something obvious I might have missed? I thought the instructions were pretty minimal for using the powershell script for 2-node HA device creation, but there was not much to be confused I thought. A few things I don't know anything about, like "ALUAOptimized" I left to "true", but interfaces, file paths, size/sector size/ etc. were all pretty self-explanatory. I am probably going to uninstall free and re-install using a trial license to see if the management UI gives me any more insight. Thanks in advance to anyone who helps!
To outline the setup:
- I have 2 Dell R710 servers, running Hyper-V 2019 (free).
- Both have a 4-port Broadcomm built-in network card, as well as a second PCIe Broadcomm 4-port card (identical chips) for a total of 8x 1gig interfaces.
- The first port of the built-in is for LAN traffic, basically so I can connect to them from LAN.
- The next 3 ports are for iSCSI/Heartbeat.
- All 4 ports of the expansion card are for Sync.
- Each port is in its own subnet between the servers, so port 2 on the integrated is 192.168.242.0/29, .1 for node 1 and .2 for node 2.
[*] I have disabled the firewall rules for the "private" network profile and forced the networks to "private", as they come up as "public" by default in windows, not sure if there is something I should be doing differently when setting up direct ethernet links so that they are not "unidentified" and classified as "public" by the OS...
[/list]
I have enabled jumbo frames on all 7 adapters used in this test, and tested/verified them with
Code: Select all
ping -f -l 8972
Code: Select all
Set-NetAdapterAdvancedProperty -RegistryKeyword "*JumboPacket" -RegistryValue 9014
Ping fails on forced fragmenting for any size greater than 8972, I suspect overhead in the packet header(s)? Not sure exactly.
Normal bandwidth seems OK; I have run iperf between the nodes over each connection and getting ~995 mbits/s with 0.001% loss with parameters like:
Code: Select all
iperf3 -c 192.168.242.1 -B 192.168.242.2 -u -b 1G -l 8972
Anyone got an idea of what I should look at next? Or if I there is something obvious I might have missed? I thought the instructions were pretty minimal for using the powershell script for 2-node HA device creation, but there was not much to be confused I thought. A few things I don't know anything about, like "ALUAOptimized" I left to "true", but interfaces, file paths, size/sector size/ etc. were all pretty self-explanatory. I am probably going to uninstall free and re-install using a trial license to see if the management UI gives me any more insight. Thanks in advance to anyone who helps!