High write latency

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Thu Nov 12, 2020 10:39 am

Hi, all!

I have successfully rolled out StarWind software on 3 nodes in my Hyper-V test environment on bare metal and it works perfectly, but I have some performance issue.
I use a SSD disk for storing an image files (one for cluster witness and second for cluster shared volume). All works perfectly, but I have a high latency for write operations.

The SSD drive has following characteristics (manufacturer):
Write latency is 65 microseconds and read latency is 55 microseconds (4 Kb block size and queue depth 1).

My experimental results is:
Write latency - 670 microseconds and read latency 55 - 57 microseconds.

I have tried place vhdx file directly on volume (not inside image file and without replication) and have got following results:
a write latency is 80 microseconds and a read latency is 55 microseconds.

I suspect that the write latency is very high.
Where can I get any information about typical latencies for StarWind software?
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Thu Nov 12, 2020 10:48 am

Greetings,

May I have logs from your setup, please? Please use this tool https://knowledgebase.starwindsoftware. ... collector/ and share the logs via Google drive here or log a case with StarWind Support and transfer the logs using this thread as a reference.
From what I see, queue depth is not significant to saturate disk performance. Please see this article for more details https://www.starwindsoftware.com/best-p ... practices/.
Did you measure the performance right on top of the HA devices presented to Windows Server as NTFS volumes? What is the network throughput? Also, what was the size of the test file?
Do you have an antivirus/firewall running in your setup?
As a workaround, you can create a HA new volume (50GB or more) and connect it to the VM over iSCSI and measure the performance inside the VM.
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Thu Nov 12, 2020 4:24 pm

Hi, Yaroslav!
I have created support request as you instructed me early. So I have collected logs and send link for downloading too.

Did you measure the performance right on top of the HA devices presented to Windows Server as NTFS volumes?
Yes. Does your software have options to place image files directly on the disk without creating volumes and formatting them?

What is the network throughput?
10 GBit for sync and 1GBit for heartbeat.

Also, what was the size of the test file?
I used iometer and didn't format volume on my vhdx file. Instead of this I used cylinder numbers that should be equal file size about 250 Gb

Do you have an antivirus/firewall running in your setup?
I haven't got any antivirus software and the firewall is off for all profiles.
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Fri Nov 13, 2020 8:30 am

Greetings,
Does your software have options to place image files directly on the disk without creating volumes and formatting them?
Yes. But we use HA images in most of our setups; it should do work just great. What you can do is presenting a test HA image directly to the VM. NOTE: it is NTFS volume, be careful with mounting it anywhere else.
I used iometer and didn't format volume on my vhdx file. Instead of this I used cylinder numbers that should be equal file size about 250 Gb
That should be sufficient. The settings though seem a little bit strange. What I would recommend is re-benchmarking everything in this way https://www.starwindsoftware.com/best-p ... practices/. Let me know if you have any questions.

I will check the logs (i.e., config) to confirm that all is fine. Will reach out to you her you early Monday or so.
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Mon Nov 16, 2020 5:04 am

I have found a couple of misconfigurations.

Only local connections. Please note that to ensure data high availability partner connections are needed.
There are not enough networks for a 3 node config.

What I would recommend is increasing queue depth while benchmarking this setup (see arbitrary diskspd setting here https://www.starwindsoftware.com/best-p ... practices/). Please also consider configuring the setup as outlined here https://www.starwindsoftware.com/resour ... rver-2016/.
Also, try connecting the StarWind HA image into the VM over iSCSI.
Try disabling caching or setting it to Write-Back mode. How to disable/change cache size https://knowledgebase.starwindsoftware. ... -l1-cache/. For SSDs, RAM cache brings moderate performance gain and normally we just leave it disabled. Try with caching disabled, please.

Please keep us posted on your performance study.
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Mon Nov 23, 2020 9:44 am

Hi Yaroslav.

Yes, the configuration uses only local connections, because high availability provided by windows failover clustering. If hard drive will not accessible on one node then VM will restart on another.
The sheet of my lab environment is below:
testlab.png
testlab.png (21.73 KiB) Viewed 5620 times
Which networks are missed?
What I would recommend is increasing queue depth while benchmarking this setup.
I dont think that it may reduce latency :). Anyway I made my experiment according to disk manufacturer parameters.
Try disabling caching or setting it to Write-Back mode.

The cache settings is write through. So we use cache for read operations. I cannot setup write-back option, because if node fails then some data will be lost.

I will study your links, but I want to get reference performance benchmarks for your software. It will help me to find the source of problem.
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Mon Nov 23, 2020 3:46 pm

high availability provided by windows failover clustering.
It provides compute resource high availability.
reference performance benchmarks for your software
Unfortunately, that is not possible as every setup has unique numbers.

On your drawing, it seems that you missed iSCSI.
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Mon Nov 23, 2020 6:53 pm

It provides compute resource high availability.
Yes, that is I need
Unfortunately, that is not possible as every setup has unique numbers.
You can publish information for few reference architectures. It will very helpful.
On your drawing, it seems that you missed iSCSI.
The iscsi layer is under vhdx file on the picture

So, I will make network tests, create ram disks on my nodes, configure HA disk on it and then run performance tests, according performance benchmarking best practices.
After that we can discuss results.
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Tue Nov 24, 2020 4:15 am

Please note that we recommend connecting the storage to the partner to avoid downtime in case of local storage failure. With partner connections, storage will be available from the partner and if there is a disk/array failure data can be accessed from partner/partners. That is what makes storage highly available. Failover clustering is more about compute resources availability and ease of management. Also, partner iSCSI connections provide better load balancing.
You can publish information for few reference architectures. It will very helpful.
I need to discuss that with my team. Will reach out to you here with more details.
So, I will make network tests, create ram disks on my nodes, configure HA disk on it and then run performance tests, according performance benchmarking best practices.
Yes, please.
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Tue Nov 24, 2020 6:24 am

Hi Yaroslav!

I've successfully installed StarWind on one of my nodes, but have one question.
How can I create ram disk from command line (I use Windows Hyper-V Server 2019)?
I found technical paper about usind StarWind ram disk, but this paper describes creation of ram disk by graphical user interface, that is not accessible in my case.
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Tue Nov 24, 2020 7:20 am

You can do that with the script in C:\Program Files\StarWind Software\StarWind\StarWindX\Samples\powershell called CreateRam.ps1.
jdeshin
Posts: 63
Joined: Tue Sep 08, 2020 11:34 am

Tue Nov 24, 2020 8:53 am

Thank you a lot!
yaroslav (staff)
Staff
Posts: 2279
Joined: Mon Nov 18, 2019 11:11 am

Tue Nov 24, 2020 10:56 am

You are always welcome. Please check PM, I shared a useful tool with you.
Post Reply