Poor Write Performance

[MilanBazant]

Hello,
I was told to post this here on forum. I tried to deploy the StarWind VSAN Free for my client in the two node variant.

I have two Windows Server 2025 hosts that use very fast NVME SSD storage each. Each uses a mirror raid of multiple of these disks. Both nodes are directly connected without any switch with 2x25Gbs NICs. I have installed StarWind software directly on these servers and created a HA disk and connected it to itself as iSCSI disk that then I used to connect to the Failover Cluster (again on the same hosts) where I want to run VMs from this disk (even this was not exactly easy as the Knowledge Base is missing the manual how to do it for Windows Appliance, Hyper-V and Free non UI version...)

The website claims that there are no performance limitations of the free version. However, I’m dealing with a very poor performance of the disk. I’m attaching two screenshots. One is the performance of the single RAID disk on one of the hosts. The second is the performance of the StarWind CSV disk. When running, I see the network cards barely utilized. Yes, I was told the CrystalDiskMark is not optimal, but still. The performance it shows is abysmal and I had to remove all VMs from such disk as the users complained of the poor performance.

11 Storage.png (242.31 KiB) Viewed 5777 times

11 SW Final.png (231.75 KiB) Viewed 5777 times

What am I doing wrong? Some advice says to increase “maxSyncQueueSize”, other say it’s not in the Free variant...
I’m also sending header of the disk and config files of my StarWind installation if that helps.

[yaroslav (staff)]

Welcome to StarWind Forum. As mentioned in DM:
-1 GB is too small to test performance for the shared storage. Please see the methodology here https://www.starwindsoftware.com/best-p ... practices/
-Use diskspd, not CrystalDiskmark. CrystalDisk patterns are very limited, and therefore, they might be capping performance.
-iSCSI itself can be a limitation. Consider trialing NVMe-oF Target (https://www.starwindsoftware.com/techni ... ew-program). You will also need StarWind NVMe-oF initiator (https://www.starwindsoftware.com/starwi ... -initiator)
StarWind NVMe-oF Target is available only for TRIAL; no plans for free deployments so far. This being said, NOT FOR PRODUCTIVE LOAD SO FAR.
FREE key cannot be armed after the technical preview.
-Technical preview systems get access to the tech teams on our side.
-No teaming please; check more in StarWind VSAN Best practices https://www.starwindsoftware.com/best-p ... practices/

We do not normally tweak maxSyncQueueSize.

Please also tell me more about
- The deployment scenario (CVM/Windows based *.exe file)
- Single disk performance
- Networking configuration
- How the iSCSI initiator is configured
- Where do you run this test (e.g., right on the host, inside the VM)?

Also, consider adding more iSCSI sessionsi in iSCSI initiator to the iSCSI target.

[MilanBazant]

Hello,
- It is installed as exe directly on the hosts. So no appliance VM atp.
- The NICs are as mentioned 2x25Gbit each. Directly connected to each other host. One network card is configured as Heartbeat and the other as Sync. Jumbo frames are enabled and I did network test and reached the 25Gbits.
- NVMe-oF Target is out of the question as you mentioned, this storage is meant for production use.
- I have a single disk performance from previous tests:

9 single disk cache.png (220.8 KiB) Viewed 5743 times

- Tests are ran directly on the hosts.
- iSCSI uses the "Microsoft", not the default option and uses MPIO.

[yaroslav (staff)]

Can you please tell me more about the storage configuration? Is it a physcal RAID array or MS Storage spaces? What RAID level you have?

- iSCSI uses the "Microsoft", not the default option and uses MPIO.

How many iSCSI sessions per iSCSI target do you have? Local and partner.

[MilanBazant]

It is a physical RAID consisting of 4 Samsung NVME SSDs, each node has 8 disks in a mirror configuration. I tested Storage Spaces extensively, but the performance of the RAID was better.

Each host has two sessions. One to each endpoint:

sess.png (56.92 KiB) Viewed 5497 times

[yaroslav (staff)]

Add more iSCSI sessions to StarWind targets.
A single iSCSI session queue cannot saturate the target performance. 3-4 sessions can.

[MilanBazant]

Do you mean adding connections to existing session?

Or else how can I add multiple sessions when I have only one NIC for iSCSI? And that NIC is 25Gbit direct host to host link that does not even register any utilization right now.

[yaroslav (staff)]

In an iSCSI initiator, you can establish multiple iSCSI sessions over a loopback and a partner network connection.
Start peeling off the layers. Try testing the performance of the iSCSI-connected NTFS volume while it is connected over just a single loopback session. Add more loopback iSCSI sesssions. See how performance changes.
Then, start adding partner iSCSI connections. See how performance changes.

Also, please use diskspd, not CrystalDiskmark, as that software does not provide means for testing according to the best practices.

[MilanBazant]

I tried to connect the device via the loopback session and now I have a disk mounted on one server (and starwind I presume syncs it to the other).

I have trouble running the test using the tool you wanted:

error Large copy.jpeg (135.2 KiB) Viewed 4173 times

[yaroslav (staff)]

That's not quite how it works.
Yes, VSAN will sync data to the replication partner. But shared storage must be connected from all replication partners; only then will client VMs benefit from shared storage.
What you can do is connect the volume over multiple iSCSI sessions and set them active through iSCSI Initiator and MPIO policy called Round robin with the subset.
Try tuning the tool. It looks like the settings are wrong; furthermore, a single thread is not enough to saturate the target performance.