Very slow synchronization

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
woorstsemml
Posts: 4
Joined: Mon Feb 08, 2021 2:51 pm

Tue Feb 16, 2021 9:33 am

Hi guys!

I´m trying to set up a 2-node HyperV Cluster with Server 2016 and I´m using the CreateHA_2.ps1 script to create the disks. The servers have identical hardware configurations: I´ve got seperate 1Gbit management, 1 Gbit heartbeat and 2x 10Gbit synchronization NICs. The underlying disk is a NVME SSD.

Now here´s what I don´t get: when I create the disk, the initial synchronization is VERY slow. I´m talking 30Kbit/s over a 10Gbit NIC. I´ve done some testing with the SSD and the 10GBit NICs and found that all of them work at their intended speeds.
Altough I can always create a tiny disk, let it synch and then resize the disk to their needed size, I´m worried that at some point the disk wants to do a full synchronization again, and then I´d have to wait a week for it to finish. Is the initial synch really that slow or did I mess up somewhere?

Here´s my CreateHA_2 Script settings:

Code: Select all

#common
	$initMethod="Clear",
	$size=20000,
	$sectorSize=512,
	$failover=0,
#primary node
	$imagePath="My computer\V\starwind",
	$imageName="SSDIMG",
	$createImage=$true,
	$storageName="",
	$targetAlias="HV02TargetSSD",
	$autoSynch=$true,
	$poolName="pool1",
	$syncSessionCount=1,
	$aluaOptimized=$true,
    #cache wb, none
	$cacheMode="none",
	$cacheSize=0,
	$syncInterface="#p2=192.199.1.30:3260,192.199.2.30:3260",
	$hbInterface="#p2=192.199.10.30:3260",
	$createTarget=$true,
#secondary node
	$imagePath2="My computer\V\starwind",
	$imageName2="SSDIMG",
	$createImage2=$true,
	$storageName2="",
	$targetAlias2="HV03PartnerSSD",
	$autoSynch2=$true,
	$poolName2="pool1",
	$syncSessionCount2=1,
	$aluaOptimized2=$false,
	$cacheMode2=$cacheMode,
	$cacheSize2=$cacheSize,
	$syncInterface2="#p1=192.199.1.20:3260,192.199.2.20:3260",
	$hbInterface2="#p1=192.199.10.20:3260",
	$createTarget2=$true
	)
Any help would be very much appreciated :mrgreen: .

Thanks in advance!
yaroslav (staff)
Staff
Posts: 2355
Joined: Mon Nov 18, 2019 11:11 am

Wed Feb 17, 2021 9:08 am

Welcome to StarWind forum!
Please check the disk performance under 64K reads. What build of StarWind VSAN do you use (Select the server -> Configuration -> Register)? Are you using Storage Spaces or do you have only one disk directly passed to the OS?
woorstsemml
Posts: 4
Joined: Mon Feb 08, 2021 2:51 pm

Wed Feb 17, 2021 3:19 pm

Thanks for the warm welcome! :)

I tested the disk read performance with diskspd with the following parameters:
diskspd.exe -b64k -d60 -o4 -t8 -w0 -c20G V:\Starwind\spd.dat
I just noticed that I mistakenly said it´s an NVME SSD, but it´s actually SATA (as you can clearly see on the data), so sorry about that. The results still max out SATA though:

Code: Select all

Input parameters:

        timespan:   1
        -------------
        duration: 60s
        warm up time: 5s
        cool down time: 0s
        random seed: 0
        path: 'V:\Starwind\spd.dat'
                think time: 0ms
                burst size: 0
                using software cache
                using hardware write cache, writethrough off
                performing read test
                block size: 65536
                using sequential I/O (stride: 65536)
                number of outstanding I/O operations: 4
                thread stride size: 0
                threads per file: 8
                using I/O Completion Ports
                IO priority: normal

System information:

        computer name: xxxx
        start time: 2021/02/17 14:58:09 UTC

Results for timespan 1:
*******************************************************************************

actual test time:       60.00s
thread count:           8
proc count:             24

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  66.88%|   1.93%|   64.95%|  33.13%
   1|  56.17%|   1.22%|   54.95%|  43.83%
   2|  58.83%|   1.80%|   57.03%|  41.17%
   3|  57.50%|   1.46%|   56.04%|  42.50%
   4|  57.99%|   1.59%|   56.41%|  42.01%
   5|  54.79%|   1.25%|   53.54%|  45.21%
   6|  58.49%|   1.30%|   57.19%|  41.51%
   7|  57.11%|   1.04%|   56.07%|  42.89%
   8|  24.09%|   1.48%|   22.60%|  75.91%
   9|   6.15%|   1.22%|    4.92%|  93.85%
  10|   0.00%|   0.00%|    0.00%| 100.00%
  11|   0.03%|   0.03%|    0.00%|  99.97%
  12|   0.00%|   0.00%|    0.00%| 100.00%
  13|   0.00%|   0.00%|    0.00%| 100.00%
  14|   0.00%|   0.00%|    0.00%| 100.00%
  15|   0.21%|   0.00%|    0.21%|  99.79%
  16|   0.00%|   0.00%|    0.00%| 100.00%
  17|   0.00%|   0.00%|    0.00%| 100.00%
  18|   0.00%|   0.00%|    0.00%| 100.00%
  19|   0.00%|   0.00%|    0.00%| 100.00%
  20|   1.54%|   0.00%|    1.54%|  98.46%
  21|   0.29%|   0.03%|    0.26%|  99.71%
  22|   0.55%|   0.00%|    0.55%|  99.45%
  23|   0.16%|   0.00%|    0.16%|  99.84%
-------------------------------------------
avg.|  20.86%|   0.60%|   20.27%|  79.14%

Total IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     37915197440 |       578540 |     602.64 |    9642.24 | V:\Starwind\spd.dat (20GiB)
     1 |     38457049088 |       586808 |     611.25 |    9780.03 | V:\Starwind\spd.dat (20GiB)
     2 |     38268305408 |       583928 |     608.25 |    9732.03 | V:\Starwind\spd.dat (20GiB)
     3 |     38471532544 |       587029 |     611.48 |    9783.72 | V:\Starwind\spd.dat (20GiB)
     4 |     38238748672 |       583477 |     607.78 |    9724.52 | V:\Starwind\spd.dat (20GiB)
     5 |     38441058304 |       586564 |     611.00 |    9775.97 | V:\Starwind\spd.dat (20GiB)
     6 |     37093965824 |       566009 |     589.59 |    9433.39 | V:\Starwind\spd.dat (20GiB)
     7 |     37318885376 |       569441 |     593.16 |    9490.59 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total:      304204742656 |      4641796 |    4835.15 |   77362.48

Read IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |     37915197440 |       578540 |     602.64 |    9642.24 | V:\Starwind\spd.dat (20GiB)
     1 |     38457049088 |       586808 |     611.25 |    9780.03 | V:\Starwind\spd.dat (20GiB)
     2 |     38268305408 |       583928 |     608.25 |    9732.03 | V:\Starwind\spd.dat (20GiB)
     3 |     38471532544 |       587029 |     611.48 |    9783.72 | V:\Starwind\spd.dat (20GiB)
     4 |     38238748672 |       583477 |     607.78 |    9724.52 | V:\Starwind\spd.dat (20GiB)
     5 |     38441058304 |       586564 |     611.00 |    9775.97 | V:\Starwind\spd.dat (20GiB)
     6 |     37093965824 |       566009 |     589.59 |    9433.39 | V:\Starwind\spd.dat (20GiB)
     7 |     37318885376 |       569441 |     593.16 |    9490.59 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total:      304204742656 |      4641796 |    4835.15 |   77362.48

Write IO
thread |       bytes     |     I/Os     |    MiB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     1 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     2 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     3 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     4 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     5 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     6 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
     7 |               0 |            0 |       0.00 |       0.00 | V:\Starwind\spd.dat (20GiB)
------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00
I use Starwind VSAN Build 13861.

No Storage Spaces in there right now, just a normal disk directly passed to the OS.
yaroslav (staff)
Staff
Posts: 2355
Joined: Mon Nov 18, 2019 11:11 am

Thu Feb 18, 2021 10:04 am

Do you have any antivirus installed? If so, please consider turning it temporarily off
woorstsemml
Posts: 4
Joined: Mon Feb 08, 2021 2:51 pm

Mon Feb 22, 2021 8:02 am

This is a pretty fresh Server 2016, so it´s only got Windows Defender. I tried disabling it temporarily and creating a new disk, but I still remain stuck at about 30Kbit when synchronizing.

I´ve also played around differentJumbo Frame Sizes (4k and 9k), but it´s also the same.
yaroslav (staff)
Staff
Posts: 2355
Joined: Mon Nov 18, 2019 11:11 am

Mon Feb 22, 2021 8:10 am

Did you try enabling TCP Offload for ip V4?
woorstsemml
Posts: 4
Joined: Mon Feb 08, 2021 2:51 pm

Mon Feb 22, 2021 10:25 am

I´ve got the following options in the NIC driver set to these values:
  • IPv4 Checksum Offload -> Rx & Tx Enabled
  • TCP Checksum Offload (IPv4) -> Rx & Tx Enabled
  • Large Send Offload V2 (IPv4) -> Enabled
yaroslav (staff)
Staff
Posts: 2355
Joined: Mon Nov 18, 2019 11:11 am

Mon Feb 22, 2021 11:48 am

Could you enable 3rd level of logging (Go to StarWind Management Console -> Select the server -> Configuration -> Advanced -> Log retention -> Log level (set to 3)) and trigger sync once again, wait for an hour, and collect the logs?
How to collect the logs https://knowledgebase.starwindsoftware. ... collector/.
Share them here with me, please, or send them to support@starwind.com. Use this thread and 467414 as a reference
thecastle2
Posts: 1
Joined: Tue Nov 15, 2022 5:26 pm

Tue Nov 15, 2022 5:29 pm

I've had the exact same problem. It happens if you had a VM running on the same drive as your trying to do an initial synch on. It dosen't affect all servers, just some, appears to be a starwind bug of some kind. Solution shutdown the VM, then starwind proceeds at a normal speed.
yaroslav (staff)
Staff
Posts: 2355
Joined: Mon Nov 18, 2019 11:11 am

Wed Nov 16, 2022 4:00 pm

Thank you for your input.
Could you please share your storage configuration? Also, if that's an ESXi deployment, what type of disk are you using? Did you change the Shares for the underlying storage of StarWind VSAN VM?
Post Reply