Writing performance not as expected

[Jan1987]

Hello,

2-node cluster:
HPE DL360 G9
P440ar RAID controller with 2GB cache and battery
6x 1TB SSD in RAID5 and 64K strip size
Formatted with 64K and NTFS
Windows Server 2022

Performance node 1 directly to the RAID volume:

Performance_Node1_Direct_RAID.jpg (75.53 KiB) Viewed 5316 times

Performance Node 2 directly to the RAID Volume:

Performance_Node2_Direct_RAID.jpg (77.94 KiB) Viewed 5316 times

Perfromance to the Starwinds vsan image on node 2:
Image was created with 512 and formatted as NTFS 4KB

Performance_Node2_Starwind_Image.jpg (74.74 KiB) Viewed 5316 times

I understand that iscsi leads to slightly weaker perfromance. the only thing that bothers me is this loss of perfromance with IO sizes between 128KB and 512KB

No replication is currently set up on the first node.
does anyone know how to fix this problem? Copying large vhdx files takes almost twice as long as when I copy it directly to the raid volume.

greetings
Jan

[yaroslav (staff)]

Hi,

Thanks for your questions.
Try adding 3x loopback and 2x partner iSCSI sessions. Also make sure write-back caching is off and you have no write-back caching on StarWind HA device's levels.
Copying is a different thing and unlike synthetic tests is bottlenecked by filling system buffers. To resolve it you need to change FirstBurstLength to 262144 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e97b-e325-11ce-bfc1-08002be10318}\000X\Parameters

[Jan1987]

Hello

sorry, unfortunately it took a little longer to try it out.

I have made a few adjustments but am not at all satisfied.

performance Node1 directly to the RAID50:

perfromance_node1.png (37.88 KiB) Viewed 2741 times

perfromance Node2 directly to RAID50 is the same because same HW Model, Same Controller and Same SSD


performance of a VM within the vSAN:

perfromance_vm.png (39.36 KiB) Viewed 2741 times

performance of a VM within vSAN (new created on the vSAN)

perfromance_vm_new.png (40.66 KiB) Viewed 2741 times

It is strange that a newly created VM has so much more performance than an existing one, but so little.
both nodes has
2x loopback
2x partner session
Registry is set on both nodes.

i have tested with HW Cache on and off, with HW cache off the write perfromance ist bad on all VMs. Cache in Starwind is set as WB and 4096MB now and the first tests was with de default 128MB i see no different in de current VM, only the new created has a lot more writeIO with this cache.
Bit VMs (old an new) have the same vhdx version, hardware version (was created on the same machine).

[yaroslav (staff)]

Can you measure performance on top of the CSV, not inside the VM?
Could you please let me know what is the network bandwidth and network topology in this clsuter?
Try more iSCSI sessions.
Also, try modifying the test parameters: it should be at least 8 threads 16 queue.
Try performance on top of NTFS volume.
Also, try a bigger file.

Copying large vhdx files takes almost twice as long as when I copy it directly to the raid volume.

This is a known problem of Windows Server 2016 and further https://learn.microsoft.com/en-us/troub ... e-transfer.
This is why I'd suggest you benchmark CSV and StarWind-based NTFS volume.

I do expect iSCSI to be the bottleneck here, but it is important to understand the NTFS and CSVFS performance. If your CSVFS volume is effectively an ReFS-based one, still try NTFS.

[Jan1987]

hi,

have tested the CSV Volume:

Screenshot 2024-01-19 174827.png (39.2 KiB) Viewed 2027 times

and here testing the vsan Volume with the image files:

Screenshot 2024-01-19 175311.png (37.44 KiB) Viewed 2027 times

for vsan the server are connected directly with 2x 10gbit an Jumbo Frames. one link for Sync an one for iSCSI. Management and VM Traffic runs over 2x 10gbit to switches.
6x loopback sessions and 6x remote session
all volumes are NTFS.

regards,
Jan

[yaroslav (staff)]

Jan,

Please make sure to have the BIOS performance profile set to high performance.
Try varying the ratio of read/write cache.
Do you see CPU overutilization during the test?
Try using Round Robin with a subset where most of the paths use partner connections (then repeat for local-only).

Will be waiting for more tests! Thanks!

[vmware]

Hi,

Thanks for your questions.
Try adding 3x loopback and 2x partner iSCSI sessions. Also make sure write-back caching is off and you have no write-back caching on StarWind HA device's levels.
Copying is a different thing and unlike synthetic tests is bottlenecked by filling system buffers. To resolve it you need to change FirstBurstLength to 262144 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e97b-e325-11ce-bfc1-08002be10318}\000X\Parameters

Can you explain what 3x loopback and 2x partner iSCSI sessions mean?More detailed, I didn't understand，thanks

[yaroslav (staff)]

In Windows, you are connecting storage over 127.0.0.1 (loopback) and partner connections.
Local sessions might have different performance.
iSCSI sessions have limit in performance and a single iSCSI session may be slower than SSD. So, by adding more of them you increase the queue of the iSCSI stack.
In a nuthsell, try adding more iSCSI sessions to storage.

[Jan1987]

Jan,

Please make sure to have the BIOS performance profile set to high performance.
Try varying the ratio of read/write cache.
Do you see CPU overutilization during the test?
Try using Round Robin with a subset where most of the paths use partner connections (then repeat for local-only).

Will be waiting for more tests! Thanks!

I have set high perfromance, is a bit better then bevor.
Round Robin with subset is enabled (default setting from Windows) an only the Loopback session are Optimize, partner session are unoptimize and in taskmanager i don't see any traffic on iSCSI Adapter, only Sync Adapter has Traffic.
CPU ist ok, about 6 to 9% when testing. here are two E5-2640V3 in the Server.

Screenshot 2024-01-20 165026.png (20.01 KiB) Viewed 1854 times

Here is a test on node1 when i stop the Starwind Service on Second Node. the same result i have when i testing on node2 and node1 is stopped.

Screenshot 2024-01-20 165530.png (33.84 KiB) Viewed 1854 times

and here a test from node 1 to node 2 (service on Node 1 stopped and all traffic goes over partner iscsi session)

Screenshot 2024-01-20 170205.png (33.47 KiB) Viewed 1854 times

with this test the Adapter shows 9.6gbits form 10gbits write and read traffic. i think the connection is good.

when the sync is enabled the perfromance drops. My idea was to use nvme in the future and 25gbit network, but when the sync is active I don't reach 10gbit.

have a nice weekend.
Jan

[yaroslav (staff)]

Hi,

That's interesting. Thanks for sharing!
Can you try multiple iSCSI sessions to the target when replication is enabled?

Can you try performance tests with a single sync adapter?
Also, can you try iSER for sync links?

[Jan1987]

Hi,

That's interesting. Thanks for sharing!
Can you try multiple iSCSI sessions to the target when replication is enabled?

Can you try performance tests with a single sync adapter?
Also, can you try iSER for sync links?

what you mean with multiple iSCSI session to the target? i have in the iscsi initator 4 session to the partner, adding more makes no different. Sync ist currently one adapter one 10gbit are dedicated for sync and the other 10gbit is dedicated for iSCSI.
iSER ist not possible, at the moment i don't have Network cards with RDMA

[yaroslav (staff)]

Since you are using round robin with subset for test, can you benchmark performance when all loopback connections are active and compare it vs partner-only connections? Try with VSAN off and on.
Try also 3x local-to-local connections. Try discovering iSCSI typing local IP (e.g., 172.16.10.10) both as target and initiator. Set those connections as preferred for round robin with subset.

I want to understand whether network-driven connections grant better performance.
Try turning off rss, VMQ, and RSC for iSCSI and Sync.

[Jan1987]

Hi,

I got it! i added a second sync adapter (used the management) and now the perfromance is doubled. looks like the Sync use only 5 to 6gbits. when i have only one sync adapter it never goes over 6Gbits, now with two adapter, both use 6Gbits.

I think i have to buy 25Gbit Cards or use both direct connected adapter for Sync AND iSCSI. the iSCSI adapter is used only when the loppback not work or the service stopped on the node.

Edit: with disabled RSS, VMQ and FlowControlle it use the full 10gbit on Sync. now i have about 1GB/s at 64KB. now my 10gbit connection is the limit need upgrade

[yaroslav (staff)]

Jan,

Great news! Seems we have resolved it.
Thanks for all your effort and time

Have a nice weekend.

[vmware]

In Windows, you are connecting storage over 127.0.0.1 (loopback) and partner connections.
Local sessions might have different performance.
iSCSI sessions have limit in performance and a single iSCSI session may be slower than SSD. So, by adding more of them you increase the queue of the iSCSI stack.
In a nuthsell, try adding more iSCSI sessions to storage.

How to add 3x lookback and 2x partner connections？ adapter NIC?
tested, after nic, there is only one session
Please tell more，thanks