Expected performance hit vs substorage

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
szohn
Posts: 4
Joined: Thu Jul 29, 2021 10:08 pm

Fri Jul 30, 2021 3:16 pm

Hello!

I'm a new user and have just setup a test environment using vsan free. It is a 2 node cluster, running windows 2019 on dell R810s. I have 3 Nics, all 10G, one for management, one for sync and one for iscsi/hearbeat. The sync and iscsi networks are direct connect without a switch Substorage is all SSD using storage spaces. The substorage virutal disk is a mirror across 6 ssds. Diskspd testing of this device using 4K blocks, I get Read 180K IOPs 750Mbs Write 120K IOPs 440Mbs

I've got the vsan setup and targets sync and appear to be working correctly, but the performance hit is of the san/iscsi layer is really significant. The best write performance I've been able to get is around 50K IOPS 240Mps.

I'm trying to find out if that is the normal overhead cost of the vsan or if I'm missing something in my configuration.

I've gone through the benchmarking white paper, my network connections are able to push >9Gps. I tried the Ram disk suggestion, using the createram.ps1 script, but that performance is worse than my ssd storage so I think I may be missing something there.

I've tried various cache options, different sizes and removing l1 cache and the changes are insignificant. I've tried testing a non replicated target from both the hosting server and the partner and performance didn't change either which seems to rule out network issues.

Can anyone tell me if that 50% performance hit is to be expected or should I keep looking for issues. And if so any suggestions on what else to try would be greatly appreciated.

Thanks,
Steve
Michael (staff)
Staff
Posts: 317
Joined: Thu Jul 21, 2016 10:16 am

Mon Aug 02, 2021 6:18 pm

Hello Steve,
Thank you for your interest in StarWind products.
Could you please clarify how exactly you are connecting the target with a non-replicated cache-less device in the ISCSI initiator (number of sessions, MPIO policy)?
Could you please double-check that Windows defender or any other antivirus apps are disabled?
szohn
Posts: 4
Joined: Thu Jul 29, 2021 10:08 pm

Tue Aug 03, 2021 1:08 pm

To be clear, I tested the following scenarios:

Replicated with L1 cache (multiple times with various size)
Replicated with no cache

These were connected using 2 sessions, localhost, and the partner IP. I tried both round robin and least depth for mpio load balancing. There was very little variation between these tests, between 50 and 60% of IOPs/Throughput using the same Diskspd tests

I also tested a non replicated image using localhost with various numbers of sessions, 1,2,4 and 8. I tried both roundrobin and least depth. The performance was the same around 55K Write iops 210Mbps vs 120K 450Mbps write on the sub storage directly.

I wasn't expecting them to be equal, but the difference is pretty significant and the fact that little changes its performance between any change makes me think i have a bottle neck I haven't identified.

AV is off
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Tue Aug 03, 2021 1:38 pm

Did you try Round-Robin with the subset with 3xlocal active paths set active?
Finally, are you experiencing this sort of performance on top of an NTFS or CSVFS volume?
Finally, did you try updating NIC drivers and firmware?
szohn
Posts: 4
Joined: Thu Jul 29, 2021 10:08 pm

Tue Aug 03, 2021 3:30 pm

I just tried the RR with subset with 3 active connections. No significant change.

All of the volumes have been straight NTFS, I have not added it to clustering yet.

I've updated the nic firmware and drivers. I can saturate them using Iperf at >9Gps.
Would the nics even come into play with a local host connection?
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Tue Aug 03, 2021 3:53 pm

No, they should not. Can I see the commands you are using to run the performance tests?
Also, did you turn off the MS Defender?
szohn
Posts: 4
Joined: Thu Jul 29, 2021 10:08 pm

Tue Aug 03, 2021 4:05 pm

I've used various, but this is my default test for writing. Windows defender is off.


diskspd -d30 -c8g -t16 -o8 -b4k -L -r -Z1G -Sh -w100 r:\spd.dat

This is the iscsi drive
diskspd -d30 -c8g -t16 -o8 -b4k -L -r -Z1G -Sh -w100 r:\spd.dat
0.00 | 0.00 | 0.000 | N/A

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 582197248 | 142138 | 18.51 | 4737.91 | 1.686 | 5.620 | r:\spd.dat (8192MB)
1 | 242110464 | 59109 | 7.70 | 1970.29 | 4.054 | 11.888 | r:\spd.dat (8192MB)
2 | 581218304 | 141899 | 18.48 | 4729.94 | 1.688 | 5.607 | r:\spd.dat (8192MB)
3 | 239198208 | 58398 | 7.60 | 1946.59 | 4.106 | 12.326 | r:\spd.dat (8192MB)
4 | 593440768 | 144883 | 18.86 | 4829.41 | 1.654 | 5.526 | r:\spd.dat (8192MB)
5 | 267247616 | 65246 | 8.50 | 2174.86 | 3.694 | 11.312 | r:\spd.dat (8192MB)
6 | 593719296 | 144951 | 18.87 | 4831.68 | 1.653 | 5.477 | r:\spd.dat (8192MB)
7 | 298287104 | 72824 | 9.48 | 2427.45 | 3.292 | 9.386 | r:\spd.dat (8192MB)
8 | 594100224 | 145044 | 18.89 | 4834.78 | 1.652 | 5.504 | r:\spd.dat (8192MB)
9 | 275410944 | 67239 | 8.76 | 2241.29 | 3.565 | 10.342 | r:\spd.dat (8192MB)
10 | 593764352 | 144962 | 18.88 | 4832.04 | 1.653 | 5.544 | r:\spd.dat (8192MB)
11 | 268103680 | 65455 | 8.52 | 2181.82 | 3.666 | 10.317 | r:\spd.dat (8192MB)
12 | 593711104 | 144949 | 18.87 | 4831.61 | 1.653 | 5.540 | r:\spd.dat (8192MB)
13 | 269799424 | 65869 | 8.58 | 2195.62 | 3.642 | 10.926 | r:\spd.dat (8192MB)
14 | 596361216 | 145596 | 18.96 | 4853.18 | 1.646 | 5.511 | r:\spd.dat (8192MB)
15 | 287326208 | 70148 | 9.13 | 2338.26 | 3.418 | 9.488 | r:\spd.dat (8192MB)
-----------------------------------------------------------------------------------------------------
total: 6875996160 | 1678710 | 218.58 | 55956.73 | 2.285 | 7.611


%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | N/A | 0.197 | 0.197
25th | N/A | 0.938 | 0.938
50th | N/A | 1.301 | 1.301
75th | N/A | 1.807 | 1.807
90th | N/A | 2.583 | 2.583
95th | N/A | 4.931 | 4.931
99th | N/A | 17.222 | 17.222
3-nines | N/A | 89.376 | 89.376
4-nines | N/A | 243.414 | 243.414
5-nines | N/A | 372.498 | 372.498
6-nines | N/A | 372.904 | 372.904
7-nines | N/A | 372.928 | 372.928
8-nines | N/A | 372.928 | 372.928
9-nines | N/A | 372.928 | 372.928
max | N/A | 372.928 | 372.928


Results from the substorage

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 839245824 | 204894 | 26.68 | 6829.91 | 1.167 | 1.793 | p:\spd.dat (8192MB)
1 | 840179712 | 205122 | 26.71 | 6837.51 | 1.166 | 1.792 | p:\spd.dat (8192MB)
2 | 838549504 | 204724 | 26.66 | 6824.25 | 1.168 | 1.793 | p:\spd.dat (8192MB)
3 | 836939776 | 204331 | 26.61 | 6811.15 | 1.170 | 1.795 | p:\spd.dat (8192MB)
4 | 936124416 | 228546 | 29.76 | 7618.33 | 1.048 | 1.707 | p:\spd.dat (8192MB)
5 | 935788544 | 228464 | 29.75 | 7615.59 | 1.048 | 1.706 | p:\spd.dat (8192MB)
6 | 935030784 | 228279 | 29.72 | 7609.43 | 1.049 | 1.711 | p:\spd.dat (8192MB)
7 | 935297024 | 228344 | 29.73 | 7611.59 | 1.049 | 1.707 | p:\spd.dat (8192MB)
8 | 935350272 | 228357 | 29.73 | 7612.03 | 1.049 | 1.708 | p:\spd.dat (8192MB)
9 | 935288832 | 228342 | 29.73 | 7611.53 | 1.049 | 1.708 | p:\spd.dat (8192MB)
10 | 935600128 | 228418 | 29.74 | 7614.06 | 1.048 | 1.708 | p:\spd.dat (8192MB)
11 | 935727104 | 228449 | 29.75 | 7615.09 | 1.048 | 1.709 | p:\spd.dat (8192MB)
12 | 935837696 | 228476 | 29.75 | 7615.99 | 1.048 | 1.707 | p:\spd.dat (8192MB)
13 | 940285952 | 229562 | 29.89 | 7652.20 | 1.043 | 1.705 | p:\spd.dat (8192MB)
14 | 935993344 | 228514 | 29.75 | 7617.26 | 1.048 | 1.707 | p:\spd.dat (8192MB)
15 | 937725952 | 228937 | 29.81 | 7631.36 | 1.046 | 1.705 | p:\spd.dat (8192MB)
-----------------------------------------------------------------------------------------------------
total: 14588964864 | 3561759 | 463.78 | 118727.29 | 1.075 | 1.728


%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | N/A | 0.074 | 0.074
25th | N/A | 0.636 | 0.636
50th | N/A | 0.976 | 0.976
75th | N/A | 1.347 | 1.347
90th | N/A | 1.686 | 1.686
95th | N/A | 1.876 | 1.876
99th | N/A | 2.235 | 2.235
3-nines | N/A | 45.723 | 45.723
4-nines | N/A | 50.205 | 50.205
5-nines | N/A | 50.790 | 50.790
6-nines | N/A | 51.160 | 51.160
7-nines | N/A | 51.841 | 51.841
8-nines | N/A | 51.841 | 51.841
9-nines | N/A | 51.841 | 51.841
max | N/A | 51.841 | 51.841
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Sat Aug 07, 2021 7:46 am

Hi there,

We had a similar problem with another user. Here is how we fixed that
1. Disabled RSS, VMQ optimization on VLANs, SRV-IO for iSCSI and Sync on both nodes.
2. Set First Burst Length under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e97b-e325-11ce-bfc1-08002be10318}\000X\Parameters to 262144.
3. Added more iSCSI sessions for the test device: 4x local and 3x partner connections. MPIO - Least Queue Depth.
4. Modified RAID controller settings: set WT and NRA as caching policies (FOR SSDs ONLY!). You can try WB and RA caching. If those are HPE boxes, set caching W/R ratio to 90/10.
Post Reply