Slow write to disk in a virtual machine

[fakir]

Good afternoon!

Sorry for my English.

There is a HYPER-V cluster configured under https://www.starwindsoftware.com/resour ... erver-2016.
Hosts - Hyper-V 2012R2 with the latest updates
StarWind Virtual SAN 8.0.0.12166
A virtual machine with Windows 8.1 Pro, also with all updates.
Network for synchronization StarWind 2 x 1Gbs
Network for VM, cluster and management - 2 x 1Gbs in the group.
For iSCSI, Multipath is used, failover policy, as the active path is 127.0.0.1.


There was a problem with the speed of writing in virtual machines. In VM, the write speed is much lower than when writing to the host disk, there are no problems with reading.

To check, I created a VHDX disk at 10gb, put it in ClusterStorage.
After
- connected the disk to the VM, diskspd issued:
Write IO
thread | bytes | I / Os | MB / s | I / O per s | AvgLat | LatStdDev | file
total: 655360 | 80 | 0.06 | 7.99 | 4240.317 | 3501.204


- mounted a disk on the host, diskspd issued:
Write IO
thread | bytes | I / Os | MB / s | I / O per s | AvgLat | LatStdDev | file
total: 178003968 | 21729 | 16.98 | 2172.92 | 16.564 | 260.102

DISKSPD ran as follows: diskspd.exe -c4G -d10 -r -w30 -t8 -o8 -b8K -h -LX: G: \ testfile.dat

If you simply write an ISO file of 3 GB size on this drive, the host speed is 80-100 MB / s, for VM 10-12 MB \ s.

Another point: if the disk is not copied to the ClusterStorage, but simply put it next to the StarWind file, and connect it to the VM. Then the write speed to this disk will be comparable with the host.

[Oleg(staff)]

Hello fakir,
Could you please give more details. Are you checking the performance of dynamic or fixed VHDX disk?
Could you please check the performance with the fixed VHDX disk?
Thank you!

[fakir]

I checked with a fixed VHDX disk. The results in Diskspd became approximately the same.
Write IO
thread | bytes | I / Os | MB / s | I / O per s | AvgLat | LatStdDev | file
total: 51691520 | 6310 | 4.93 | 630.97 | 48.718 | 191.741

At the same time, the sequential recording from the VM has increased, but still much less than on the host. Host 90-100, VM 15-20 MB / s

Server:
HP ML350 G6 \ 4 x SAS 300GB 10k (in raid 10).

The result of diskspd for the partition on which the StarWind file is located:

Write IO
thread | bytes | I / Os | MB / s | I / O per s | AvgLat | LatStdDev | file
total: 88604672 | 10816 | 8.45 | 1081.60 | 0.108 | 0.127

[Oleg(staff)]

Could you please share underlying storage performance test results from both servers on the same patterns?

[fakir]

The first server (previous tests were done on it)
HP ML350 G6, 4 x SAS 300 GB 10k in RAID 10

Code: Select all

C:\temp\diskspd\amd64fre>diskspd.exe -c4G -d30 -r -w30 -t8 -o8 -b8K -h -LX: D:\testfile.dat

Command Line: diskspd.exe -c4G -d30 -r -w30 -t8 -o8 -b8K -h -LX: D:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 30s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'D:\testfile.dat'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing mix test (read/write ratio: 70/30)
                block size: 8192
                using random I/O (alignment: 8192)
                number of outstanding I/O operations: 8
                thread stride size: 0
                threads per file: 8
                using I/O Completion Ports
                IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time:       30.00s
thread count:           8
proc count:             24

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   3.85%|   1.25%|    2.60%|  96.15%
   1|   3.96%|   1.51%|    2.45%|  96.04%
   2|   3.13%|   1.46%|    1.67%|  96.88%
   3|   1.72%|   0.83%|    0.89%|  98.28%
   4|   0.94%|   0.31%|    0.63%|  99.06%
   5|   1.46%|   0.16%|    1.30%|  98.54%
   6|   2.55%|   0.26%|    2.29%|  97.45%
   7|   1.56%|   0.21%|    1.35%|  98.44%
   8|   0.00%|   0.00%|    0.00%| 100.00%
   9|   0.00%|   0.00%|    0.00%| 100.00%
  10|   0.00%|   0.00%|    0.00%| 100.00%
  11|   0.00%|   0.00%|    0.00%| 100.00%
  12|   3.75%|   1.77%|    1.98%|  96.25%
  13|   2.34%|   0.94%|    1.41%|  97.66%
  14|   1.77%|   0.94%|    0.83%|  98.23%
  15|   0.57%|   0.31%|    0.26%|  99.43%
  16|   0.00%|   0.00%|    0.00%| 100.00%
  17|   0.10%|   0.00%|    0.10%|  99.90%
  18|   0.42%|   0.00%|    0.42%|  99.58%
  19|   0.00%|   0.00%|    0.00%| 100.00%
  20|   0.00%|   0.00%|    0.00%| 100.00%
  21|   0.00%|   0.00%|    0.00%| 100.00%
  22|   0.00%|   0.00%|    0.00%| 100.00%
  23|   0.00%|   0.00%|    0.00%| 100.00%
-------------------------------------------
avg.|   1.17%|   0.41%|    0.76%|  98.83%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       110272512 |        13461 |       3.51 |     448.70 |   17.838 |    29.121 | D:\testfile.dat (4096MB)
     1 |       106602496 |        13013 |       3.39 |     433.77 |   18.444 |    30.821 | D:\testfile.dat (4096MB)
     2 |       107339776 |        13103 |       3.41 |     436.77 |   18.296 |    29.284 | D:\testfile.dat (4096MB)
     3 |       108756992 |        13276 |       3.46 |     442.53 |   18.064 |    29.409 | D:\testfile.dat (4096MB)
     4 |       108552192 |        13251 |       3.45 |     441.70 |   18.133 |    29.602 | D:\testfile.dat (4096MB)
     5 |       108085248 |        13194 |       3.44 |     439.80 |   18.193 |    29.875 | D:\testfile.dat (4096MB)
     6 |       109936640 |        13420 |       3.49 |     447.33 |   17.867 |    29.515 | D:\testfile.dat (4096MB)
     7 |       107782144 |        13157 |       3.43 |     438.57 |   18.230 |    29.892 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:         867328000 |       105875 |      27.57 |    3529.17 |   18.131 |    29.691

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        76881920 |         9385 |       2.44 |     312.83 |   25.537 |    31.946 | D:\testfile.dat (4096MB)
     1 |        74489856 |         9093 |       2.37 |     303.10 |   26.348 |    33.942 | D:\testfile.dat (4096MB)
     2 |        75522048 |         9219 |       2.40 |     307.30 |   25.959 |    31.949 | D:\testfile.dat (4096MB)
     3 |        75931648 |         9269 |       2.41 |     308.97 |   25.827 |    32.235 | D:\testfile.dat (4096MB)
     4 |        75694080 |         9240 |       2.41 |     308.00 |   25.957 |    32.472 | D:\testfile.dat (4096MB)
     5 |        74792960 |         9130 |       2.38 |     304.33 |   26.243 |    32.855 | D:\testfile.dat (4096MB)
     6 |        75939840 |         9270 |       2.41 |     309.00 |   25.818 |    32.506 | D:\testfile.dat (4096MB)
     7 |        75300864 |         9192 |       2.39 |     306.40 |   26.048 |    32.804 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:         604553216 |        73798 |      19.22 |    2459.94 |   25.965 |    32.591

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        33390592 |         4076 |       1.06 |     135.87 |    0.110 |     0.107 | D:\testfile.dat (4096MB)
     1 |        32112640 |         3920 |       1.02 |     130.67 |    0.109 |     0.049 | D:\testfile.dat (4096MB)
     2 |        31817728 |         3884 |       1.01 |     129.47 |    0.107 |     0.045 | D:\testfile.dat (4096MB)
     3 |        32825344 |         4007 |       1.04 |     133.57 |    0.106 |     0.156 | D:\testfile.dat (4096MB)
     4 |        32858112 |         4011 |       1.04 |     133.70 |    0.109 |     0.129 | D:\testfile.dat (4096MB)
     5 |        33292288 |         4064 |       1.06 |     135.47 |    0.108 |     0.040 | D:\testfile.dat (4096MB)
     6 |        33996800 |         4150 |       1.08 |     138.33 |    0.107 |     0.089 | D:\testfile.dat (4096MB)
     7 |        32481280 |         3965 |       1.03 |     132.17 |    0.105 |     0.023 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:         262774784 |        32077 |       8.35 |    1069.23 |    0.108 |     0.092


  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      0.067 |      0.064 |      0.064
   25th |      6.579 |      0.093 |      0.113
   50th |     14.554 |      0.102 |      7.277
   75th |     32.637 |      0.113 |     22.772
   90th |     62.035 |      0.129 |     49.682
   95th |     88.011 |      0.142 |     74.069
   99th |    157.084 |      0.184 |    141.059
3-nines |    276.615 |      0.701 |    256.728
4-nines |    441.528 |      4.480 |    422.099
5-nines |    631.886 |      9.463 |    547.021
6-nines |    631.886 |      9.463 |    631.886
7-nines |    631.886 |      9.463 |    631.886
8-nines |    631.886 |      9.463 |    631.886
9-nines |    631.886 |      9.463 |    631.886
    max |    631.886 |      9.463 |    631.886

Second server
HP ML350 G5, 6 x SATA 1TB 7.2k in RAID 10

Code: Select all

C:\temp\diskspd\amd64fre>diskspd.exe -c4G -d30 -r -w30 -t8 -o8 -b8K -h -LX: D:\testfile.dat

Command Line: diskspd.exe -c4G -d30 -r -w30 -t8 -o8 -b8K -h -LX: D:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 30s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'D:\testfile.dat'
                think time: 0ms
                burst size: 0
                software cache disabled
                hardware write cache disabled, writethrough on
                performing mix test (read/write ratio: 70/30)
                block size: 8192
                using random I/O (alignment: 8192)
                number of outstanding I/O operations: 8
                thread stride size: 0
                threads per file: 8
                using I/O Completion Ports
                IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time:       30.01s
thread count:           8
proc count:             8

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   5.21%|   0.47%|    4.74%|  94.78%
   1|   4.48%|   0.83%|    3.65%|  95.50%
   2|   3.65%|   0.89%|    2.76%|  96.34%
   3|   2.50%|   0.36%|    2.14%|  97.48%
   4|   2.55%|   0.94%|    1.61%|  97.48%
   5|   2.81%|   0.68%|    2.14%|  97.17%
   6|   3.44%|   0.94%|    2.50%|  96.55%
   7|   2.81%|   0.68%|    2.14%|  97.17%
-------------------------------------------
avg.|   3.43%|   0.72%|    2.71%|  96.56%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        22978560 |         2805 |       0.73 |      93.48 |   85.674 |    87.152 | D:\testfile.dat (4096MB)
     1 |        22716416 |         2773 |       0.72 |      92.42 |   86.709 |    88.664 | D:\testfile.dat (4096MB)
     2 |        22708224 |         2772 |       0.72 |      92.38 |   86.728 |    87.085 | D:\testfile.dat (4096MB)
     3 |        22970368 |         2804 |       0.73 |      93.45 |   85.654 |    85.274 | D:\testfile.dat (4096MB)
     4 |        23085056 |         2818 |       0.73 |      93.92 |   85.356 |    86.926 | D:\testfile.dat (4096MB)
     5 |        23216128 |         2834 |       0.74 |      94.45 |   84.852 |    87.660 | D:\testfile.dat (4096MB)
     6 |        22896640 |         2795 |       0.73 |      93.15 |   85.997 |    88.624 | D:\testfile.dat (4096MB)
     7 |        22290432 |         2721 |       0.71 |      90.68 |   88.265 |    87.964 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:         182861824 |        22322 |       5.81 |     743.94 |   86.143 |    87.426

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        16089088 |         1964 |       0.51 |      65.46 |  106.752 |    84.878 | D:\testfile.dat (4096MB)
     1 |        15867904 |         1937 |       0.50 |      64.56 |  108.645 |    86.962 | D:\testfile.dat (4096MB)
     2 |        16080896 |         1963 |       0.51 |      65.42 |  107.441 |    84.849 | D:\testfile.dat (4096MB)
     3 |        16220160 |         1980 |       0.52 |      65.99 |  105.692 |    83.253 | D:\testfile.dat (4096MB)
     4 |        16203776 |         1978 |       0.52 |      65.92 |  105.720 |    84.774 | D:\testfile.dat (4096MB)
     5 |        16572416 |         2023 |       0.53 |      67.42 |  105.052 |    86.665 | D:\testfile.dat (4096MB)
     6 |        15720448 |         1919 |       0.50 |      63.96 |  106.783 |    86.415 | D:\testfile.dat (4096MB)
     7 |        15966208 |         1949 |       0.51 |      64.96 |  109.094 |    86.279 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:         128720896 |        15713 |       4.09 |     523.68 |  106.883 |    85.524

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |         6889472 |          841 |       0.22 |      28.03 |   36.451 |    71.053 | D:\testfile.dat (4096MB)
     1 |         6848512 |          836 |       0.22 |      27.86 |   35.885 |    69.684 | D:\testfile.dat (4096MB)
     2 |         6627328 |          809 |       0.21 |      26.96 |   36.468 |    70.352 | D:\testfile.dat (4096MB)
     3 |         6750208 |          824 |       0.21 |      27.46 |   37.506 |    69.333 | D:\testfile.dat (4096MB)
     4 |         6881280 |          840 |       0.22 |      28.00 |   37.404 |    71.764 | D:\testfile.dat (4096MB)
     5 |         6643712 |          811 |       0.21 |      27.03 |   34.462 |    67.528 | D:\testfile.dat (4096MB)
     6 |         7176192 |          876 |       0.23 |      29.19 |   40.463 |    75.373 | D:\testfile.dat (4096MB)
     7 |         6324224 |          772 |       0.20 |      25.73 |   35.679 |    67.959 | D:\testfile.dat (4096MB)
-----------------------------------------------------------------------------------------------------
total:          54140928 |         6609 |       1.72 |     220.26 |   36.832 |    70.508


  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      0.185 |      0.231 |      0.185
   25th |     43.370 |      0.283 |     11.684
   50th |     88.272 |      0.356 |     66.923
   75th |    149.723 |     49.281 |    131.005
   90th |    227.076 |    135.311 |    211.079
   95th |    265.047 |    194.894 |    255.681
   99th |    357.010 |    296.922 |    347.556
3-nines |    723.215 |    408.738 |    659.273
4-nines |    827.449 |    448.339 |    824.457
5-nines |    833.672 |    448.339 |    833.672
6-nines |    833.672 |    448.339 |    833.672
7-nines |    833.672 |    448.339 |    833.672
8-nines |    833.672 |    448.339 |    833.672
9-nines |    833.672 |    448.339 |    833.672
    max |    833.672 |    448.339 |    833.672

[Oleg(staff)]

You have two servers with different underlying storage configuration and different performance.
As you have StarWind HA on top of this configuration, the performance is limited with the speed of the slower side, 6 x SATA 1TB 7.2k in RAID 10.