Page 1 of 1
Very slow synchronization
Posted: Tue Feb 16, 2021 9:33 am
by woorstsemml
Hi guys!
I´m trying to set up a 2-node HyperV Cluster with Server 2016 and I´m using the CreateHA_2.ps1 script to create the disks. The servers have identical hardware configurations: I´ve got seperate 1Gbit management, 1 Gbit heartbeat and 2x 10Gbit synchronization NICs. The underlying disk is a NVME SSD.
Now here´s what I don´t get: when I create the disk, the initial synchronization is VERY slow. I´m talking 30Kbit/s over a 10Gbit NIC. I´ve done some testing with the SSD and the 10GBit NICs and found that all of them work at their intended speeds.
Altough I can always create a tiny disk, let it synch and then resize the disk to their needed size, I´m worried that at some point the disk wants to do a full synchronization again, and then I´d have to wait a week for it to finish. Is the initial synch really that slow or did I mess up somewhere?
Here´s my CreateHA_2 Script settings:
Code: Select all
#common
$initMethod="Clear",
$size=20000,
$sectorSize=512,
$failover=0,
#primary node
$imagePath="My computer\V\starwind",
$imageName="SSDIMG",
$createImage=$true,
$storageName="",
$targetAlias="HV02TargetSSD",
$autoSynch=$true,
$poolName="pool1",
$syncSessionCount=1,
$aluaOptimized=$true,
#cache wb, none
$cacheMode="none",
$cacheSize=0,
$syncInterface="#p2=192.199.1.30:3260,192.199.2.30:3260",
$hbInterface="#p2=192.199.10.30:3260",
$createTarget=$true,
#secondary node
$imagePath2="My computer\V\starwind",
$imageName2="SSDIMG",
$createImage2=$true,
$storageName2="",
$targetAlias2="HV03PartnerSSD",
$autoSynch2=$true,
$poolName2="pool1",
$syncSessionCount2=1,
$aluaOptimized2=$false,
$cacheMode2=$cacheMode,
$cacheSize2=$cacheSize,
$syncInterface2="#p1=192.199.1.20:3260,192.199.2.20:3260",
$hbInterface2="#p1=192.199.10.20:3260",
$createTarget2=$true
)
Any help would be very much appreciated
.
Thanks in advance!
Re: Very slow synchronization
Posted: Wed Feb 17, 2021 9:08 am
by yaroslav (staff)
Welcome to StarWind forum!
Please check the disk performance under 64K reads. What build of StarWind VSAN do you use (Select the server -> Configuration -> Register)? Are you using Storage Spaces or do you have only one disk directly passed to the OS?
Re: Very slow synchronization
Posted: Wed Feb 17, 2021 3:19 pm
by woorstsemml
Thanks for the warm welcome!
I tested the disk read performance with diskspd with the following parameters:
diskspd.exe -b64k -d60 -o4 -t8 -w0 -c20G V:\Starwind\spd.dat
I just noticed that I mistakenly said it´s an NVME SSD, but it´s actually SATA (as you can clearly see on the data), so sorry about that. The results still max out SATA though:
Code: Select all
Input parameters:
timespan: 1
-------------
duration: 60s
warm up time: 5s
cool down time: 0s
random seed: 0
path: 'V:\Starwind\spd.dat'
think time: 0ms
burst size: 0
using software cache
using hardware write cache, writethrough off
performing read test
block size: 65536
using sequential I/O (stride: 65536)
number of outstanding I/O operations: 4
thread stride size: 0
threads per file: 8
using I/O Completion Ports
IO priority: normal
System information:
computer name: xxxx
start time: 2021/02/17 14:58:09 UTC
Results for timespan 1:
*******************************************************************************
actual test time: 60.00s
thread count: 8
proc count: 24
CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 66.88%| 1.93%| 64.95%| 33.13%
1| 56.17%| 1.22%| 54.95%| 43.83%
2| 58.83%| 1.80%| 57.03%| 41.17%
3| 57.50%| 1.46%| 56.04%| 42.50%
4| 57.99%| 1.59%| 56.41%| 42.01%
5| 54.79%| 1.25%| 53.54%| 45.21%
6| 58.49%| 1.30%| 57.19%| 41.51%
7| 57.11%| 1.04%| 56.07%| 42.89%
8| 24.09%| 1.48%| 22.60%| 75.91%
9| 6.15%| 1.22%| 4.92%| 93.85%
10| 0.00%| 0.00%| 0.00%| 100.00%
11| 0.03%| 0.03%| 0.00%| 99.97%
12| 0.00%| 0.00%| 0.00%| 100.00%
13| 0.00%| 0.00%| 0.00%| 100.00%
14| 0.00%| 0.00%| 0.00%| 100.00%
15| 0.21%| 0.00%| 0.21%| 99.79%
16| 0.00%| 0.00%| 0.00%| 100.00%
17| 0.00%| 0.00%| 0.00%| 100.00%
18| 0.00%| 0.00%| 0.00%| 100.00%
19| 0.00%| 0.00%| 0.00%| 100.00%
20| 1.54%| 0.00%| 1.54%| 98.46%
21| 0.29%| 0.03%| 0.26%| 99.71%
22| 0.55%| 0.00%| 0.55%| 99.45%
23| 0.16%| 0.00%| 0.16%| 99.84%
-------------------------------------------
avg.| 20.86%| 0.60%| 20.27%| 79.14%
Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 37915197440 | 578540 | 602.64 | 9642.24 | V:\Starwind\spd.dat (20GiB)
1 | 38457049088 | 586808 | 611.25 | 9780.03 | V:\Starwind\spd.dat (20GiB)
2 | 38268305408 | 583928 | 608.25 | 9732.03 | V:\Starwind\spd.dat (20GiB)
3 | 38471532544 | 587029 | 611.48 | 9783.72 | V:\Starwind\spd.dat (20GiB)
4 | 38238748672 | 583477 | 607.78 | 9724.52 | V:\Starwind\spd.dat (20GiB)
5 | 38441058304 | 586564 | 611.00 | 9775.97 | V:\Starwind\spd.dat (20GiB)
6 | 37093965824 | 566009 | 589.59 | 9433.39 | V:\Starwind\spd.dat (20GiB)
7 | 37318885376 | 569441 | 593.16 | 9490.59 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total: 304204742656 | 4641796 | 4835.15 | 77362.48
Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 37915197440 | 578540 | 602.64 | 9642.24 | V:\Starwind\spd.dat (20GiB)
1 | 38457049088 | 586808 | 611.25 | 9780.03 | V:\Starwind\spd.dat (20GiB)
2 | 38268305408 | 583928 | 608.25 | 9732.03 | V:\Starwind\spd.dat (20GiB)
3 | 38471532544 | 587029 | 611.48 | 9783.72 | V:\Starwind\spd.dat (20GiB)
4 | 38238748672 | 583477 | 607.78 | 9724.52 | V:\Starwind\spd.dat (20GiB)
5 | 38441058304 | 586564 | 611.00 | 9775.97 | V:\Starwind\spd.dat (20GiB)
6 | 37093965824 | 566009 | 589.59 | 9433.39 | V:\Starwind\spd.dat (20GiB)
7 | 37318885376 | 569441 | 593.16 | 9490.59 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total: 304204742656 | 4641796 | 4835.15 | 77362.48
Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
1 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
2 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
3 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
4 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
5 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
6 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
7 | 0 | 0 | 0.00 | 0.00 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00
I use Starwind VSAN Build 13861.
No Storage Spaces in there right now, just a normal disk directly passed to the OS.
Re: Very slow synchronization
Posted: Thu Feb 18, 2021 10:04 am
by yaroslav (staff)
Do you have any antivirus installed? If so, please consider turning it temporarily off
Re: Very slow synchronization
Posted: Mon Feb 22, 2021 8:02 am
by woorstsemml
This is a pretty fresh Server 2016, so it´s only got Windows Defender. I tried disabling it temporarily and creating a new disk, but I still remain stuck at about 30Kbit when synchronizing.
I´ve also played around differentJumbo Frame Sizes (4k and 9k), but it´s also the same.
Re: Very slow synchronization
Posted: Mon Feb 22, 2021 8:10 am
by yaroslav (staff)
Did you try enabling TCP Offload for ip V4?
Re: Very slow synchronization
Posted: Mon Feb 22, 2021 10:25 am
by woorstsemml
I´ve got the following options in the NIC driver set to these values:
- IPv4 Checksum Offload -> Rx & Tx Enabled
- TCP Checksum Offload (IPv4) -> Rx & Tx Enabled
- Large Send Offload V2 (IPv4) -> Enabled
Re: Very slow synchronization
Posted: Mon Feb 22, 2021 11:48 am
by yaroslav (staff)
Could you enable 3rd level of logging (Go to StarWind Management Console -> Select the server -> Configuration -> Advanced -> Log retention -> Log level (set to 3)) and trigger sync once again, wait for an hour, and collect the logs?
How to collect the logs
https://knowledgebase.starwindsoftware. ... collector/.
Share them here with me, please, or send them to
support@starwind.com. Use this thread and 467414 as a reference
Re: Very slow synchronization
Posted: Tue Nov 15, 2022 5:29 pm
by thecastle2
I've had the exact same problem. It happens if you had a VM running on the same drive as your trying to do an initial synch on. It dosen't affect all servers, just some, appears to be a starwind bug of some kind. Solution shutdown the VM, then starwind proceeds at a normal speed.
Re: Very slow synchronization
Posted: Wed Nov 16, 2022 4:00 pm
by yaroslav (staff)
Thank you for your input.
Could you please share your storage configuration? Also, if that's an ESXi deployment, what type of disk are you using? Did you change the Shares for the underlying storage of StarWind VSAN VM?