Slow performance on one of two servers

[WolfR1der]

We have two identical servers set up with Starwinds iSCSI and running shared cluster storage. In testing after discovering the slow speed it is not RAID array related. When copying a large file from server to server over a 1GB link any server copying to the server in question the average speed of the copy is 112MBps. This is being copied to a non-Starwind volume. When I get onto that specific server if I copy the file from one location to another on the same drive I get over 700MBps copy speed. However, once I try to copy this file to the Starwinds iSCSI volume wildly various speeds. On start of the file copy there's a pause for a few seconds where no data moves then it finally spikes to 115MBps before finally settling down to around 11-22MBps and most of the copy speed hangs out around 11-33MBps from there on out though occasionally it will bottom out at 0.

Server2 does not exhibit any of this behavior. The same kinds of copies to a Starwinds directory will average around 175MBps which isn't great considering internally these servers can hit over 700MBps but it's far better than 11-33MBps.

Any suggestions?

[Boris (staff)]

Do I get your case right?
1. Copying over 1Gbit network is fine and up to the limit of the network itself (non-StarWind disk tested).
2. Copying to the StarWind iSCSI disk shows you different figures, but mainly goes at about 11-33 MBps.
If so, how is your StarWind iSCSI disk connected to the host? What is the MPIO policy used? How many sessions are there for the target? What is the type of storage and RAID underneath StarWind? I would appreciate more information about your configuration, as that will definitely be helpful for troubleshooting.

[WolfR1der]

You are correct.

Windows Server 2016
Xeon E5-2630v4
192GB
RAID6/Read Policy:Always Read Ahead/IO Policy:Cached IO/Write Policy:WriteBack/Access Policy:Read Write
12x WDC WD4002FYYZ0 hard drives

I followed one of Starwind's own how to articles on setting these servers up. They are set up identically. Each iSCSI device MPIO is set up for Least Queue Depth and is connected via loopback at 127.0.0.1. Microsoft iSCSI initiator is also the adapter listed in the details. Also each Starwinds device is 1250GB with 1.25GB RAM cache.

[Oleg(staff)]

Could you please check performance with the help of testing tools?
Try to measure disk performance with diskspd or fio utility.

[WolfR1der]

After finally finding time to reboot this machine the performance problem has not cleared.

Do you have a particular set of parameters you want me to use with diskspd? I am working with it now and will post some results soon.

Edit: The issue has not completely cleared but I have to acknowledge that it has in fact gotten better since the reboot. I am guessing there was synching at the time that was killing performance but I am now getting between 70 and 11 MB/s on file copies through Windows Explorer.

[Michael (staff)]

Hello!

Please find below the list of the commands for diskspd.

Note, you should create C:\test folder before the test. It is assuming that you are testing D drive. Number of threads 10 is equal to number of cores.

diskspd.exe -t10 -b4K -r -w0 -o20 -d60 -h -L -c50G D:\test.io > c:\test\4k_random_readD_HDD.txt

diskspd.exe -t10 -b4K -r -w100 -o20 -d60 -h -L -c50G D:\test.io > c:\test\4k_random_writeD_HDD.txt

diskspd.exe -t10 -b64K -r -w0 -o20 -d60 -h -L -c50G D:\test.io > c:\test\64k_random_readD_HDD.txt

diskspd.exe -t10 -b64K -r -w100 -o20 -d60 -h -L -c50G D:\test.io > c:\test\64k_random_writeD_HDD.txt

diskspd.exe -t10 -si -b4K -w0 -o20 -d60 -h -L -c50G D:\test.io > c:\test\4k_seq_readD_HDD.txt

diskspd.exe -t10 -si -b4K -w100 -o20 -d60 -h -L -c1000G D:\test.io > c:\test\4k_seq_writeD_HDD.txt

diskspd.exe -t10 -si -b64K -w0 -o20 -d60 -h -L -c1000G D:\test.io > c:\test\64k_seq_readD_HDD.txt

diskspd.exe -t10 -si -b64K -w100 -o20 -d60 -h -L -c1000G D:\test.io > c:\test\64k_seq_writeD_HDD.txt