ESXi 5.1 performance poor, MS iSCSI initiator fast

[Tijz]

I now had the opportunity to test on a different switch.

But still got the same results.
I mounten a VMDK on the SAN.
From the VM I ran IOMeter with 100% reads using 16K blocks within 1GB (2048000 sectors) file.

I get a 15MB/s consistent read speed.

I did the same for the NFS test with the same results.

[Tijz]

Ok update:

I now ran IOMeter with same settings directly on the Starwind server.
I got about 70MB/s

then using the MS Windows iSCSI initiator ON THE STARWIND SERVER, I connected to 127.0.0.1.

I ran IOMeter again on the iSCSI volume. Now I got only 20MB/s!

So Starwind does seem the bottleneck.
What can I change?

[Max (staff)]

Could you please clarify - which test gives you 70 mbs?
Is it your local hard drive or is it a test of an iSCSI device attached to the localhost?

[Tijz]

Could you please clarify - which test gives you 70 mbs?
Is it your local hard drive or is it a test of an iSCSI device attached to the localhost?

I ran IOMeter on the machine where I installed Starwind on.
I did this so see if my storage itself wasn't the problem. I got 70MB/s. Not astronomical, but fair for a couple of SATA disks in RAID5.
I than did the same test over iSCSI on the local server. So I connected to 127.0.0.1.
Than I got only 20MB/s.
So same storage, same server, but now using iSCSI (through starwind software)
strange right?

[Max (staff)]

Ok got it.
That's really strange,
Was it a RAM disk or an image file?
If that was an image file - did you use any cache on the StarWind device?

Also, I didn't find it in the previous posts - did you have a chance to install SW on another machine and run the same VMware benchmark?

[Tijz]

It is an image file.
Cache in Starwind is configured as write back.

But i don't have any battery backed up cache on my hardware, so the hardware storage controller is write through (so not using write cache).

Unfortunately I don't have other hardware to install starwind on.

edit: just a plane old image file, no dedupe or compression or whatever.

[Max (staff)]

It is an image file.
Cache in Starwind is configured as write back.
But i don't have any battery backed up cache on my hardware, so the hardware storage controller is write through (so not using write cache).

StarWind uses server's RAM for caching so it is a little different from the controller cache.
There is a tiny chance that caching interferes with the performance in this case.
In order to check this you'd to recreate the image file device without any cache and run a benchmark against it.
For a solid confirmation - a RAM disk test on iSCSI localhost (should result in ~70 MB/s) and to any other destination (100-117MB/s depending on the adapter)

[Tijz]

Thanks for your reply.

I added a new image file and set "no-caching".
Still I'm getting only 20MB/s throughput. I also tried with the op option "asynchronous" disabled, but no change.
From the RAM disk i got 30MB/s.

Noteworthy though is that i'm only using one worker thread (for all tests described in this topic). When I add a worker thread, my performance doubles. Adding another one, and I get 60MB/s

But when using one workerthread directly on the disk (so not using iSCSI) I get 70MB/s

EDIT: Same goes for tests from VM's using VMDK's on the Starwind server. Adding worker threads, doubles performance, only here by increments of about 12MB/s

[Tijz]

When I add about 10 worker thread in a VM, I get 75MB/s throughput.

Was I just testing wrong? I would expect when copying a single file to get the maximum available throughput, but that's not happening.
However, the throughput is available when copying multiple files.

There is still something wrong right?

[Max (staff)]

Wait a minute,
And what is the # of outstanding IOps set to?

[Tijz]

this is set to 1.

And yes, I changed it to 10 and now I get 70+ MB/s oops..

But... It all began when I copied one 2GB file to VMDK hosted bij Starwind.
My 1Gbps netwerkconnection only gets used by 15% (on the starwind server)
It still does this...
VMWare performance monitor shows the same.

So that's when I started testing with IOMeter, which i failed to setup correctly apparently...

But how to explain the slow copy?
To test I used iSCSI initiator of Microsoft from another physical host and copied the same file. I got almost 100% utilized of my 1Gbps link.
that's why I made my topic title "ESXi 5.1 performance poor, MS iSCSI initiator fast".

[Max (staff)]

So now the question is - why are we not getting good performance on ESXi under low loads.
Previously I've encountered similar case, but the link utilization was decreasing over time. So 100 first 10 minutes and getting worse then.
Also, just curious - did you try to do the same procedure using storage vMotion?

[Tijz]

Yes, I very much would like to know that too.

Sadly I don't have storage vmotion, it's two free edition stand alone hosts at that site.


I'm currently in the process of setting up another starwind server in our own test environment to see I get similar results.

[Tijz]

I just finished testing on our own environment.

Now I get 1Gpbs througput. ESX and windows version are the same, the hardware is completely different.


Could this really be hardware issue?

The environment with the problem does use VMWare certified hardware, but it's a clone server (SuperMicro).

EDIT: does it matter on what kind of storage the Starwind software is installed on? It's installed on slow storage (the program itself, not the image file it's exporting, that's stored on fast array)

EDIT: also, when I run IOMeter in my new test environment, with only 1 oustanding I/O, I get about 60MB/s. So there's really something wrong with the other environment. but what?

[Max (staff)]

Well, if you're not able to reproduce this behavior on another hardware I would blame hardware.
Other thing is I don't like to blame hardware without knowing the exact reason (but that's another story)
StarWind itself is sitting in memory by the time it's running, so doesn't matter if you install it on a usb stick or on SSD.
1 Outstanding IO does not really show real load on iSCSI. It may be a difference in array logics (local storage is optimized for any load vs iSCSI optimized for bigger load)