Poor read performance from vsphere 5

HostingTeam · Mon Aug 19, 2013 4:35 pm

Hi. I am trying version 6 of the iSCSI SAN free edition.

But when I am testing the performance I get poor read performance.
Write performance is wire speed.

I have been reading the forum but without luck and in a dead end.
Please help.

lohelle · Mon Aug 19, 2013 4:43 pm

Remember that all writes can be cached, and that would give wirespeed until the cache is full, but for reads data will have to be accessed from the underlying storage (all non-cached data).

You could try to disable Delayed ACK in the initiator, but I think write speed usually is hurt from that (sometimes).

HostingTeam · Mon Aug 19, 2013 5:21 pm

I know.
Locally on the SAN I have 300 MB/sec Read and 250 MB/sec write performance.

On the Virtuel machine I have 110 MB/sec write and 30 MB/sec read.
The read performance is OK when the transfer size is below 64 KB

Disabling the ACK make the read performance a little bit worse.

lohelle · Mon Aug 19, 2013 5:36 pm

Try creating a small ramdisk and check performance on that. Just to have something more to go on.

If using MPIO, have you tried using a single path? Jumbo frames vs no jumbo frames?

HostingTeam · Mon Aug 19, 2013 5:52 pm

Same performance when using a ramdisk.
I am not using Jumboframes and I am using a single path.

lohelle · Mon Aug 19, 2013 6:04 pm

Just throwing out ideas here.

Have you tried using a different nic on the host and/or on the Starwind server? Maybe you could test with a RAMdisk on a different computer.
If you have a unused nic on the vSphere host, you might want to try a direct connection to rule out switch problems.
I'm not saying that sone equipment is not working right, but sometimes computer/software problems are not logical..

If your switches have some ISCSI optimization, try disabling that btw.

HostingTeam · Mon Aug 19, 2013 6:16 pm

Great ideas. But without luck.

I have previosly tried different SAN hardware, different ESX hardware, different NIC and different switches...

lohelle · Mon Aug 19, 2013 6:38 pm

Very strange.
There MUST be something special/different with your setup, as this is not happening with me/others. Tried other cables?

Have you tried benchmarking using MS ISCSI initiator in a VM?

HostingTeam · Mon Aug 19, 2013 7:32 pm

yes.

Then I get wirespeed. I get 100 MB/sec write and 100 MB/sec read.

Mon Aug 19, 2013 9:06 pm

Nice. What do you use to check the performance? Can you run ATTO Disk Benchmark and post screenshots here? Thank you!

HostingTeam · Tue Aug 20, 2013 3:40 pm

I use ATTO.
I have attached 2 screenshots.

Tue Aug 20, 2013 4:23 pm

It's definitely something related to the packet size as sub-64KB reads are fine. Let us take a closer look @ the issue...

HostingTeam · Tue Aug 20, 2013 4:49 pm

Are you able to help ?

Tue Aug 20, 2013 6:12 pm

Yes absolutely! Techies will jump into this soon.

HostingTeam wrote:Are you able to help ?

jtmroczek · Thu Aug 22, 2013 11:12 pm

Tagging post to see outcome. We saw similar behavior with Windows initiators during bench-marking.In our tests, the problem seemed to worsen as the number of MPIO paths increased. For us, performance is good enough even with the degraded performance. We punted on the issue in the hopes microsoft or starwind would address the issue before we needed the performance on large blocks. We plan on retesting when V8 hits the streets.