vSAN free on proxmox losing management network after reboot

[mb1977]

Hi.
I have a virtual appliance for proxmox running vsan free - just starting out.
I have3 NIC's on the VM - one virtual NIC from proxmox and 2 pci passthrough interfaces, and I plan to use the passthrough NICs for iscsi (two vlans for multipathing).
I set everything up, just fine, assigns the interfaces its' roles, and everything is working. Until i reboot the appliance.
After reboot, one of the passthrough iscsi NICs is suddenly becoming the management interface, and the only way to access the webgui is to hook up a vm to the iscsi vlan and access the webgui on its iscsi IP and correct the role assignments from there.

Is there a way to enforce a NIC to be management NIC?

I tried to play around with the scripts in /opt/starwind/starwind-network-management without success.

starwindnics.png (24.03 KiB) Viewed 70190 times

[yaroslav (staff)]

Hi,
Welcome to StarWind Forum!
I think there is an issue with the pass-through functions and the adapter. Can you try without pass-through to see if that is the case?

[mb1977]

Hi.
Yes, that will work, but then I'll only haveone network adapter.
I could include the nics in the virtualization layer and add two more virtual adapters and try.

I'll be back with results.
Thanks.

[mb1977]

When I remove all adapters except for the management adapter, it works fine (but then I'm not able to use iscsi because I lack the NICs for it obviously).

When I add two more virtualized adapters, there is only a blank screen on the webgui telling me: "Management service is starting. Please wait..." after starting up.

Attached screenshot is what appears when i go into the console and try to set the management network manually.

vsandump.png (173.16 KiB) Viewed 70175 times

[mb1977]

After a bit of fiddling with the NIC's I think I got it working.

First I added both iscis interfaces, but the mgmt GUI wouldn't start. The interfaces are now virtualized with a bridge in proxmox.

Then I removed one of them, and created a netplan entry for the remaining one, enabled the interface by ifup. rebooted and checked the GUI - fine.

After that, I added the other iscsi NIC with the same procedure, and after a few retries and reboots, it looks good.

[yaroslav (staff)]

Thank you for your update.
I noticed this happening to SRIOV: network functions (meant to be a "listener" adapter) can get conflicting settings with the "main" or a different adapter *.yaml.
Glad to know that you fixed it. Thanks for your update.
Please make sure not to use network abstractions for production scenarios. See more on StarWind VSAN best practices https://www.starwindsoftware.com/best-p ... practices/

[treeedge]

This is exactly the problem I am having however, I have checked and rebooted following your updates. This problem has been fixed. So grateful for this!

[mb1977]

OK, so it's working, but I'm not happy with running an abstraction layer on the iscsi nets (for the same reasons it's not recommended for production use).
Also, it takes a bit of throughput from the performance when I put a VM on it, compared to the passthrough solution.

So, If virtualized nics are not recommended for production, and passhtrough nics is a problem - is there anything I can do?

We like the product, and we'd be interested in paying for it if it fits the bill, but as of now, I understand that I really can't use the product?

best regards
--
Markus

[yaroslav (staff)]

Could you please let me what NICs you use and what is the storage? It might happen that they are not the bottleneck. You can use the product as is. May I also wonder where is the recommendation not to use virtualized NICs?
The problem is with the way OS (Ubuntu, just as any other OS) works with the passthrough functions. We are planning to improve this in the next builds.
You can also log a support call with us at support@starwind.com to resume cooperation on this case.

[mb1977]

The NICs are Intel X710 for 10GbE SFP+, connected with DAC cables to a Nexus switch.

The storage is Dell 0GYGVV, SATA SSD 1.92TB where the appliance sees the disks directly, RAID 50 with 2 spares, 24 disks in the storage pool. No replication, just a simple volume.

17.46 TB raw capacity on the pool
16 TB capacity on the volume
15.62 TB capacity on the LUN

Multipath IO (round-robin) towards ESX hosts.

The vsan appliance have 16GB of RAM and 16vcpus.

The idea of not using virtualized NIC's is that it adds an extra layer of software between the nic and the appliance, but if it's OK for production, it's fine by me.

[yaroslav (staff)]

Thank you for your udpate.
You can use VirtIO, it will cut some of the network performance, but I doubt that RAID50 is capable of saturating 10 GbE on random writes anyway.
By the way, are you using the hardware RAID or Software RAID?
Still, may I ask you to log a case with support@starwind.com to look into the pass-through problem together?

[mb1977]

I'm using built-in software RAID.
When using the passthrogh NIC's I had a bit of better throughput on the VM side, so it would be nice to be able to use it.

I've logged a case with support, thanks for your help so far.

Thank you for your udpate.
You can use VirtIO, it will cut some of the network performance, but I doubt that RAID50 is capable of saturating 10 GbE on random writes anyway.
By the way, are you using the hardware RAID or Software RAID?
Still, may I ask you to log a case with support@starwind.com to look into the pass-through problem together?

[yaroslav (staff)]

Thanks for your cooperation. I have asked my colleagues to take care of the case.
Thanks!