So today I updated the network drivers/VIBs within ESXi for the adapter (nmlx5-core) with the most recent ones from Mellanox's website and disabled DRSS/RSS mode as recommended in the release notes for vSphere 6.7. This helped a little bit and I was able to get 20Gb/s with iperf directly from ESXi host to ESXi host.
We have two identical hosts with this hardware/config:
HPE DL380 Gen9
2x Intel 2660 v3 2.6GHz
128 GB DDR4 2133 (HP certified)
HPE 640FLR-SFP28 25Gb/s Mellanox ethernet (firmware version 14.23.8052, driver version 4.17.14.2)
HPE P440ar storage controller (firmware version 6.88)
5x HPE 1.2tb write-intensive SSD's in RAID5
1x 250gb Virtual Disk for StarWind VM and other local VMs
1x 100gb Virtual Disk passed directly to StarWind VM for image storage
1x 100gb Virtual Disk passed directly to test VM
I'm using the most recent StarWind VSA image for vSphere. Storage for the VSA's StarWind Images are stored on 100gb virtual disks that were made in the raid controller and passed through to the VM using "New RAW disk" in ESXi. The disks were zero formatted as xfs file systems and mounted to the /mnt/disk1 directory. I then proceeded to use the StarWind Management Console to create a 40gb Hard Disk/Virtual Disk device without any RAM or flash cache and replicated the image to the second host. Sync and Heartbeat/iSCSI channels are set to the two separate VMXNET3 interfaces that run through separate vSwitches to each of the 640FLR-SFP28's 25Gb/s interfaces.
I also set the setting 'Disk.DiskMaxIOSize' on each ESXi host to 512, added the Dynamic iSCSI targets in VMware, and created a 40gb VMFS6 datastore from the STARWIND device presented through iSCSI.
To test actual disk performance, I spun-up a Win10 Pro virtual machine with its hard disk located on the 40gb datastore that runs through StarWind's iSCSI interface to the ESXi hosts. Using CrystalDiskMark 6, these were my results. C: is stored on the 40gb HA image through StarWind iSCSI, E: is a RAW 100gb virtual disk passed directly to the VM from the raid controller. Nothing else was being preformed on the server when these tests were being done.
StarWind disk:
RAW Virtual Disk (direct storage controller access):
The same results occur when using Window's winsat.
StarWind disk:
RAW Virtual Disk (direct storage controller access):
What is the issue? Obviously something is causing latency/speed issues, but the network still runs at a minimum 15Gb/s. Any suggestions?