vSphere 3-node VSAN & direct connect vMotion

[primo]

Hello,

does vMotion work in the configuration with only direct connections to each host if multiple vMotion VMKernels are configured (https://kb.vmware.com/s/article/2052092)?

NOTE: It is recommended to set jumbo frames to 9000 on vSwitches and VMKernel ports for iSCSI and Synchronization traffic. Additionally, vMotion can be enabled on VMKernel ports.
https://www.starwindsoftware.com/resour ... phere-6-5/

Thank you!

[yaroslav (staff)]

Hi,

Yes, you can enable vMotion on direct links. You can enable it for iSCSI.

[primo]

Thanks! How do I overcome the situation that the wrong VMKernel port is used when vMotion is enabled on 2 VMKs per server?

Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout. Migration [167837953:8412176475878412281] failed to connect to remote host <10.1.3.2> from host <10.1.2.1>: Network unreachable. vMotion migration [167837953:8412176475878412281] vMotion migration [167837953:8412176475878412281] stream thread failed to connect to the remote host <10.1.3.2>: The ESX hosts failed to connect over the VMotion network The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration. vMotion migration [167837953:8412176475878412281] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout

Node 1:
VMK1: 10.1.1.1/24
VMK2: 10.1.2.1/24

Node 2:
VMK1: 10.1.1.2/24
VMK2: 10.1.3.2/24

Node 3:
VMK1: 10.1.2.3/24
VMK2: 10.1.3.3/24

[yaroslav (staff)]

Just select all iSCSI interfaces on all 3 servers. Please be aware that this action drops connectivity over the selected for a moment. In other words, there is a good risk of bringing down the production if done on all links too fast (i.e., without rescanning after enabling the feature on one VMK).

[primo]

I did, but when doing vMotion, I get the above error.
On one server, vMotion selects VMK with IP 10.1.3.2 is selected, on another the IP 10.1.2.1. Because the link is wrong (no connection) and subnets don't match, it fails.

Thanks!

[yaroslav (staff)]

Are you sure that VMK settings (security and other tabs) are the same across 3 servers?
Please try VMKPINGing the iSCSI adapters during vMotion.

[primo]

Yes, the settings are the same across all VMKs:

Security
Promiscuous mode Reject
MAC address changes Reject
Forged transmits Reject
Traffic shaping
Average bandwidth --
Peak bandwidth --
Burst size --

VMKPing also works.
I've also tested VM migration by leaving vMotion enabled only on one VMKernel port (I've manually turned off vMotion for the remaining non-relevant VMK).

ESX1 -> ESX2
ESX2 -> ESX1

ESX1 -> ESX3
ESX3 -> ESX1

ESX2 -> ESX3
ESX3 -> ESX2

This was all successful. Is there any special configuration required because of this article https://kb.vmware.com/s/article/2052092?

[yaroslav (staff)]

On StarWind Side, no. Just avoid using VLANs for the iSCSI and Sync if those are connected directly.

[primo]

In your lab, did vMotion work when enabled on 2 VMKernels per server?
If this doesn't work, it's maybe a good idea to update the white paper with the information that for a 3-node VSAN it's necessary to use a switch for vMotion.

Thanks!

[yaroslav (staff)]

Thanks for the request. Will discuss that with my team.
UPD: Yes, you are right. You can use either management network or a dedicated VMKernel.