iSER_DM is not loaded

[SiegfriedB]

I am trying to get iSER to work in the free Windows version. On both nodes, I am getting these errors at the start of the StarWind vSAN service.

Code: Select all

2/13 21:25:54.103663 1780 General: DllMain: NetDirect providers are not found.
2/13 21:25:54.103797 1780 Srv: *** iScsiServer::iScsiServer: iSER_DM is not loaded.

I believe it should work on that system. The network adapters in that server are

• Intel E810-XXVDA2
• Broadcom BCM57416

Both of these should support iSER but it shows N/A in the GUI for every adapter.



I have tried so far:

• updating the network drivers to the latest version for each adapter
• I set up listeners in the config file

  Code: Select all

  <iSerListen value="192.168.51.2,192.168.52.2,192.168.53.2,192.168.54.2,192.168.55.2,192.168.56.2"/>

• installed NVM update for Intel adapters
• Set Network Direct Functionality to Enabled in the driver settings for all adapters
• Set all adapters to RoCEv2 in the "Network Direct Technology" driver settings. The default for Intel is iWARP.
• Ensured that Windows has enabled RDMA for all adapters (Enable-NetAdapterRdma -Name "*").

  Code: Select all

  get-netadapterRdma
  
  Name                      InterfaceDescription                     Enabled     PFC        ETS
  ----                      --------------------                     -------     ---        ---
  Ethernet 4                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 8                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 3                Supermicro 10GBASE-T Ethernet Controlle… True        False      True
  Ethernet 9                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 6                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 7                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 5                Intel(R) Ethernet Network Adapter E810-… True        False      True
  Ethernet 2                Supermicro 10GBASE-T Ethernet Controller True        False      True

What could this be? I'm on free Hyper-V Server 2019.

[yaroslav (staff)]

Out of my experience, both x810 and Broadcom NICs are full of "surprises" with RDMA. iSER in StarWind VSAN does not work over iWARP.
Please stop the service on each side (make sure fast sync completes in between), and disable rename the iSER_DM.dll.
At this point, iSER is deprecated pretty much, it didn't take off on Windows or anywhere else.
We're close to release NVMe-oF (RDMA & TCP) in StarWind VSAN CVM.

[SiegfriedB]

Hi, thank you for your reply. I found the problem and fixed it! I'm putting it here in case someone needs it in the future.

At this point, iSER is deprecated pretty much, it didn't take off on Windows or anywhere else.
We're close to release NVMe-oF (RDMA & TCP) in StarWind VSAN CVM.

Will you make NVMe-oF available in the free version when it works in the CVM? I wonder which provides better performance in the current state, Win app with iSER or CVM without iSER/NVMe-oF.

Solution

1. Flow Control
   I tested if RDMA works with Microsoft's Test-Rdma.ps1 script and it did not work. I got an error saying flow control is disabled. I activated flow control in the driver settings for all adapters and ran the script again.
   
   I found this: Intel recommends flow control "strongly" for RoCEv2.
   Flow Control in RDMA Networks - 1.3 - ID:635330 | Intel® Ethernet 800 Series Linux Flow Control
   
   
   The default setting is Disabled. I set it to Auto Negotiation.
   
   
   Now, RDMA worked in this test.
   
2. NetDirect user mode RDMA provider
   Now, this error in vSAN made me wonder. Is there a NetDirect provider that is missing in the operating system? (Note, this is my first time working with RDMA.)

   Code: Select all

   2/13 21:25:54.103663 1780 General: DllMain: NetDirect providers are not found.

   After a little research, I found that an RDMA user mode provider is needed to enable applications to use RDMA. It is on the ISO in Intel's driver download.
   location: Release_28.3\APPS\PROSETDX\Winx64\DxSetup.exe
   
   
   After this, I rebooted both nodes.
   
   

[yaroslav (staff)]

Thanks for your feedback. NVMe-oF, from my experience grants way better performance than iSCSI and iSER. That's why I am excited to hear users' feedback! It should be available soon.
Speaking of RDMA, I learned one thing recently. They have different MTU values (check the Network Direct Max Transmission parameter, if I remember correctly). If misaligned with the target, packet fragmentation might increase CPU utilization.

Thanks again for sharing your experience!!

[yaroslav (staff)]

We have a guide that should minimize the chance of downtime if done correctly https://knowledgebase.starwindsoftware. ... d-version/.
For CVM, press the Gear button at the top -> Update. The general prerequisites: check storage connectivity, check HA devices synchronization, etc. are the same.

[SiegfriedB]

Thanks for your feedback. NVMe-oF, from my experience grants way better performance than iSCSI and iSER. That's why I am excited to hear users' feedback! It should be available soon.

Yes, but I would like to know if you will make it available in the free version once it is included in CVM.
As for right now, I am curious about the performance but I have no way of testing NVMe-oF. My hardware supports it but my vSAN trial license expired before I was able to do any testing. I signed up in October 2023 for a free and a trial license but my server has only now been delivered.
If I understand it correctly, CVM currently does not have NVMe-oF and the Windows version of vSAN does have it but it is impossible to configure in the free version (if my research is correct).

Speaking of RDMA, I learned one thing recently. They have different MTU values (check the Network Direct Max Transmission parameter, if I remember correctly). If misaligned with the target, packet fragmentation might increase CPU utilization.

Thanks again for sharing your experience!!

I found out about this too, yesterday.
The Intel driver offers two settings:

• Jumbo Packet with possible values: Disabled (default), 4088 bytes and 9014 bytes.
• ROCEv2 Frame Size. Possible values: 1024, 2048, 4096

Broadcom has

• Jumbo Packet and it lets me freely set any value between 1514 and 9174.
• Network Direct MTU. Possible values: 256, 512, 1024, 2048, 4096

Why should I trust StarWind's software update process and what measures do they have in place to ensure data integrity and system stability during the update?

Why do you hijack my thread? Your question is way off topic. I suggest you create your own post.

[yaroslav (staff)]

Siegfried,

If I understand it correctly, CVM currently does not have NVMe-oF and the Windows version of vSAN does have it but it is impossible to configure in the free version (if my research is correct).

Thanks for your question!
StarWind Free's philosophy is to provide the product with little to no restrictions as compared to the paid version. I do not think we will restrict FREE users from NVMe-oF (hope that vision does not change ). True, scripts may need to be reworked a bit, so the feature might be available for FREE users slightly later. For TRIAL users, though, NVMe-oF transport might be available sooner as they have access to UI. I have asked dev team to look into it beforehand, so I hope it will not get delayed .

Again, thanks for sharing your experience.
P.s. Let me know if your trial key expired and if you need the new one.
p.p.s . You cannot switch from TRIAL to FREE.

[SiegfriedB]

P.s. Let me know if your trial key expired and if you need the new one.
p.p.s . You cannot switch from TRIAL to FREE.

Yes, please, I would like to get a new trial key as my first trial key expired without me using it. I requested the trial key in October 2023 and I am only now in the position to test features like NVMe-oF or iSER. Back in October my research had not progressed to a point where I knew what kind of hardware I needed. I did some testing in a virtual environment back then and I never activated/used the trial key.

StarWind Free's philosophy is to provide the product with little to no restrictions as compared to the paid version. I do not think we will restrict FREE users from NVMe-oF (hope that vision does not change ). True, scripts may need to be reworked a bit, so the feature might be available for FREE users slightly later. For TRIAL users, though, NVMe-oF transport might be available sooner as they have access to UI. I have asked dev team to look into it beforehand, so I hope it will not get delayed .

That is good to hear. I noticed some "restrictions" in the free version that made me believe that it wasn't put in on purpose as a restriction and that, rather, the restriction comes from the PowerShell cmdlets being incomplete. For instance, it is impossible to choose the sector size (512B or 4K) for a RAM drive in PS but that is possible in the GUI.

[yaroslav (staff)]

Hi,

Yes, please, I would like to get a new trial key as my first trial key expired without me using it.

Please reach out to us at support@starwind.com use this thread and 1120081 as your references.

I requested the trial key in October 2023 and I am only now in the position to test features like NVMe-oF or iSER.

NVMe-oF is still on its way and for CVMs only. Stay tuned.
iSER seems to be abandoned down the road.