switch from Hyper-V to ESX

[Jan1987]

Hello,

I am currently considering switching from my current Hyper-V setup with vSAN Free to ESX.

Unfortunately, Hyper-V does not make sense for a home lab in the long term because the trial versions expire and a license costs a lot of money.
Since I have access to NFR licenses of vSphere via the company and VMUG, this would make more sense.

I have also already tried to set up the vSAN on the ESX, but I am far from the performance I had with Hyper-V. To me it looks like iSCSI is the problem again.

About the setup:
2x DL360 G9
6x1TB SSD (RAID50) per server
2x 500GB (RAID1) per server
2x 10gbit (Management & VM) per server
2x 10Gbit (Sync & iSCSI) per server

If I install a VM on the RAID50, it has over 3000MB/s write and 2000MB/s write (sequential). I have the same performance when I assign the RAID50 as RAW device to the VM.

I tried it with CVM and a Windows VM. As soon as I present the StarWind disk via iscsi to the ESX host, I only have around 500MB/s write and 300MB/s write.

I had the same problem with Hyper-V and it can be easily solved with several iscsi sessions. But with ESX I can only set up one?
i am aware that i am limited to the 10gbit but i don't know if this also applies to the vswitch. with Windows iSCSI i was able to use the full 10gbit.

best regards,
Jan

[yaroslav (staff)]

I am currently considering switching from my current Hyper-V setup with vSAN Free to ESX.

Unfortunately, Hyper-V does not make sense for a home lab in the long term because the trial versions expire and a license costs a lot of money.

That's strange given the Broadcom moves and the "end" of Free ESXi.

iSCSI is the problem again.

It is. Try using more iSCSI sessions, i.e., try using more VMkernels, and port groups, and discover the targets over iSCSI more IPs therefore. SRIOV here could help too.
P.S. NVMe-oF should not have this problem. Stay tuned, it should be a part of CVM soon.

[Jan1987]

That's strange given the Broadcom moves and the "end" of Free ESXi.

Microsoft also no longer offers Hyper-V free of charge since Server 2022.
either I only use Server 2019 or switch to vmware where I have NFR keys.

It is. Try using more iSCSI sessions, i.e., try using more VMkernels, and port groups, and discover the targets over iSCSI more IPs therefore. SRIOV here could help too.

I have made 3 vmkernels with different IP addresses, but in the same range. I don't know how ESX splits this up but the performance is unfortunately just as bad as if I only use one VMkernel
I have change Disk.DiskMaxIOSize to 512 and checked the I/O Schedule, all correct and SRIOV is enabled in the BIOS for the Network Card. in ESX i don't find an option.

Screenshot 2024-03-29 163335.png (170.37 KiB) Viewed 5504 times

Screenshot 2024-03-29 163301.png (231.28 KiB) Viewed 5504 times

P.S. NVMe-oF should not have this problem. Stay tuned, it should be a part of CVM soon.

is here a ETA and is it supported in esx?

[yaroslav (staff)]

Hi,

1 GB file is too small use something bigger. Also, it is important to understand underlying storage performance numbers.
The ETA should be roughly 1 month (optimistic). Stay tuned.

[Jan1987]

Thanks, I have given up my attempt with ESX.

No matter how many VMKernels I make and no matter if they are different networks the maximum speed in iscsi is just 1 to 2 gbit/s

I have the same problem with a Windows VM and Starwind on it. So I will wait for NVME-of for CVM and hope that this works under ESX.

I now have my old setup again directly via the Windows hosts and it runs perfectly. Does NVME-of work under Windows and vSAN Free? Maybe I can get a little more out of this, especially with small IO I am a little dissatisfied.

best regards,
Jan

[yaroslav (staff)]

You need CVM for NVMe-oF.
NVMe-oF, unlike iSCSI, is designed slightly differently and allows maxing out performance with one session. I think that is a problem with VMware: a single iSCSI session that bottlenecks the shared storage performance.