Weird Perfomance Problem

[MarkH]

Using a standard vmware setup to the starwind target (in eval mode), I'm seeing very peculiar behavior. When I copy large files to the iSCSI target (an image file), I get excellent copy performance (near wire-speed over 1GbE) -- right up until the point that about 2GB has been transferred. From that point on the transfer rate slows to a crawl, dropping from 80-90MB/sec down to 1-4MB/sec and stays there until the copy has finished. Sometimes, in fact, the copy never finishes (or perhaps I didn't wait long enough -- but the xfer rate was showing something close to 0KB/s). This behavior occurs whether I copy 1 big file >2GB or several smaller files that total more than 2GB.

My setup is:

Storage server is an Asus Z8NA-DC6 motherboard; dual quad-core Intel 5504 CPU's, 2.0 GHz; 16GB RAM; 3Ware 9690SA 8-port SAS/SATA controller; 2x74 SAS 15K drives in a RAID 1 mirror for the system disk; 6X1TB Seagate Barracuda drives in a RAID 6 configuration (3.7TB useable) for my data drive. The motherboard has a dual-port Intel 82574L-based 1GbE adapter, and an Intel PRO/1000PT dual-port 1GbE card in a PCI-E slot.

VMware host is a Dell 1950 dual quad-core Intel 5405 CPU, 2.0 GHz; 16GB RAM; Dell PERC-6 controller with 2x1TB Seagate Barracuda drives in a RAID 1 mirror for the system drive. The sytems has dual-port Broadcom BCM5708-based 1GbE adapters in one PCI-E slot, and there is an additional dual-port Intel PRO/1000ET 1GbE adapter in the other PCI-E slot.

The iSCSI target is exposed as 192.168.200.182 on one of the Intel adpaters, and the VMWare host has a vmkernel port configured on 192.168.200.82. No other hosts are on this network, and only iSCSI traffic is presented to the vmKernel port on the vmware host. The two devices are connected via a Dell PowerConnect 5324 managed switch. I'm using a standard sized Ethernet frame and MTU.

I'm completely baffled. What could be going on here?

Thanks,

-M.

[Constantin (staff)]

I`ll ask our QA team, but I purpose that possible reason of it - your RAID controller. I`ll update you with new info ASAP.

[MarkH]

I'm beginning to have the same suspicion. I've created a 4GB RAMDisk starwind target and the problem doesn't occur there. I can copy files >2GB with no performance issues -- the copy runs, as it should, at pretty close to wire-speed.

-M.

[anton (staff)]

Did you play with the disk attached locally? I mean running ATTO Disk Benchmark & Intel I/O Meter shows expected numbers or no?

I'm beginning to have the same suspicion. I've created a 4GB RAMDisk starwind target and the problem doesn't occur there. I can copy files >2GB with no performance issues -- the copy runs, as it should, at pretty close to wire-speed.

-M.

[MarkH]

So the main problem turned out to be simple: I didn't have write cache enabled on the controller. Before I discovered that I ran the Atto disk benchmark on the storage server itself (no iSCSI, no VMWare -- just direct attached storage. I got the results in the first (a screenshot from Atto). The write performance leaves a bit to be desired ....

No write cache

FileserverJoel-RAID6.png (27.24 KiB) Viewed 4418 times

After discovering that write cache on the controller was disabled, I turned it on and immediately saw an improvement in the benchmark, as shown in the second image.

Write cache enabled

FileserverJoel-RAID6-wcache.png (26.56 KiB) Viewed 4416 times

...which looks a lot better.

In real life, though, things didn't improve much at all. After attaching a VMWare-provisioned iSCSI drive to my test VM I used the standard MS Windows 2008 drag-and-drop functionality to copy a couple of hundred GB of .iso images to the iSCSI drive from a network drive (the network drive was accessed via a completely separate NIC from the one serving the iSCSI disk). Copy performance starts out great -- right at 100MB/sec for a GB or so, then declines to about 8MB/sec for the remainder of the copy. The copy completes, eventually (VERY eventually) -- but never speeds up beyond 8-9MB/sec.

I'm getting an Adaptec 5805Z controller to play with tomorrow -- I'll report back my results from that.

-M.

[anton (staff)]

If you use an SMB share do you have better write performance?