Cannot connect Windows Server 2022 to ONTAP9 NVMe/TCP Target

[mark2.tech]

Hi there. We're having issues with the Initiator failing to 'Connect' using NVMe/TCP.
Host: Windows Server 2022 with second adapter in VLAN11: 192.168.11.6
Target: NetApp ONTAP 9.15 with dedicated VLAN11 Broadcast Domain, IP Space, and NICs on each Node.
All Systems have Jumbo Frames enabled.
I changed the Initiator name to 'nqn' in the 'Configuration' Tab.

Screenshot_20241210-NQN.png (11.11 KiB) Viewed 2548 times

In ONTAP, it would not accept the nqn that windows was using, so it had to be
nqn.2008-08.com.starwind:gallatin.domain.local
vs. windows that would not discover the target unless it was set to
nqn.2008-08.com.starwind.gallatin.domain.local
I added the two Target Portals successfully 192.168.11.204:4420 and 192.168.11.205:4420 using the iSCSI initiator properties tool.

Screenshot_20241210-Discovery.png (12.92 KiB) Viewed 2548 times

When I click the 'Targets' Tab I see the Discovered Targets and status is inactive. The target name is:
"tcp:nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery"
When I attempt to connect To Target, I get the following error: 'Log On to Target: "Login Failed". I've tried specifying the Advanced settings, including the 'Local adpater as 'StarWind NVMe over Fabrics Storage Controller' and the 'Initiator IP:'as the dedicated VLAN11 adapter and the Target portal IP: as the 192.168.11.204/8009

Screenshot_20241210-LoginFailed.png (16.88 KiB) Viewed 2548 times

Running the command from elevated Powershell:

StarNVMeoF_Ctrl.exe discovery_tcp 192.168.11.204:4420 192.168.11.6

I can see the two Discovery nodes

Code: Select all

StarNVMeoF Controller Application v 2.0.0.672
Our Miniport #2 (ver 2.0 build 672)
Discovering of TCP addr 192.168.11.204:4420 from 192.168.11.6...
        NET_LUID:       0x6008003000000
Discovery returned: gen 6, numrec 2
1) subnqn nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery
        Discovery: port 0x2, cntlid 0xffff, trtype TCP, addr 192.168.11.205 : 8009, adrfam 1, asqsz 32
2) subnqn nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery
        Discovery: port 0x1, cntlid 0xffff, trtype TCP, addr 192.168.11.204 : 8009, adrfam 1, asqsz 32

Running

StarNVMeoF_Ctrl.exe insert_all_tcp 192.168.11.204:4420 192.168.11.6

I get failures:

Code: Select all

StarNVMeoF Controller Application v 2.0.0.672
Our Miniport #2 (ver 2.0 build 672)
Discovering of TCP addr 192.168.11.204:4420 from 192.168.11.6...
        NET_LUID:       0x6008003000000
Discovery returned: gen 6, numrec 2
Connecting all TCP devices ...
        NET_LUID:       0x6008003000000
Connecting to TCP target at addr 192.168.11.205:8009 from 192.168.11.6 SubNQN=nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery HostNQN=nqn.2008-08.com.starwind.gallatin nq=0 qd=0 node=0 core=0...
STAR_NVMEOF_IOCTL_TARGET_ADD request failed (err 0, status 0xc00000bb)!
        NET_LUID:       0x6008003000000
Connecting to TCP target at addr 192.168.11.204:8009 from 192.168.11.6 SubNQN=nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery HostNQN=nqn.2008-08.com.starwind.gallatin nq=0 qd=0 node=0 core=0...
STAR_NVMEOF_IOCTL_TARGET_ADD request failed (err 0, status 0xc00000bb)!

Let me know if you need anything else to assist.
Thank you in advance.

[yaroslav (staff)]

Hi,

Sorry for the delay.
Do you have NVMeoF support enabled on NetApp side?
Please also share with me
1. update levels of Windows server
2. what NICs and drivers do you use?
3. Are storage and Windows Server connected directly?
4. Any firewall/antivirus enabled? Windows Defender counts.
5. CHAP or any authentication settings on NetApp side?

[mark2.tech]

Hi yaroslav,

1. update levels of Windows server : Fully patched Server 2022
2. what NICs and drivers do you use? Virtual Machine using Hyper-V Network Adapter
3. Are storage and Windows Server connected directly? No, they're connected via Meraki MS350 switch-stack with trunked ports specific to the VLAN11 we're using.
4. Any firewall/antivirus enabled? Windows Defender counts. SentinelOne is in place with Windows Defender Firewall.
5. CHAP or any authentication settings on NetApp side? No CHAP or authentication is enabled.

Thank you

[yaroslav (staff)]

Can I ask to uninstall the AV software and reboot it?
So is it a VM? May I wonder what is the host OS and what is the underlying hardware NIC?
Also, is it possible to connect NetApp to the Windows Server host using StarWind NVMe-oF Initiator?

[mark2.tech]

Can I ask to uninstall the AV software and reboot it? Done - This did not change the errors.
So is it a VM? May I wonder what is the host OS and what is the underlying hardware NIC? Yes a VM. Underlying NIC is a Broadcom NetXtreme Gigabit Ethernet adapter. I see Driver version 214.0.0.3.
Also, is it possible to connect NetApp to the Windows Server host using StarWind NVMe-oF Initiator? I can probably bypass the Switch for testing purposes, but in production, we'll need to operate via switch.

[yaroslav (staff)]

I think the main use case was exposing NIC to the initiator. Try SR-IOV or passing the adapter into the VM somehow (could help https://cloudinfrastructureservices.co. ... -it-works/)
From my experience, Broadcom may be very funny in terms MTU (try 1500 or 1514 on the entire network stack) and RDMA (does not seem relevant).
I will be happy to take a closer look at the initiator running on top of the Hyper-V host. Log a call with us if it does not work on Windows Server bare-metal.

[mark2.tech]

So run NVMe/TCP can be run without Jumbo frames at 1500MTU?
So are you saying that the Initiator doesn't work with VMs? I have enabled SR-IOV, rebuild my virtual switch in Hyper-V and tried again with no change to behavior
It would seem odd to me since iSCSI works perfectly fine via Hyper-V Virtual NIC.
The error is a Login issue, is that related to the NIC? Are there Log files we can use to determine why it is claiming a login failed?

[yaroslav (staff)]

So are you saying that the Initiator doesn't work with VMs?

I am saying that Hyper-V switches are not intended use as far as I know and we prefer fewer virtualization layers.

So run NVMe/TCP can be run without Jumbo frames at 1500MTU?

I have enabled SR-IOV, rebuild my virtual switch in Hyper-V and tried again with no change to behavior

Try on the physical host, please.

[mark2.tech]

Hi there. I was able to reconfigure the testing to be directly on the Hyper-V Host, so we no longer have to work with the VM Layers for now. and we're working directly on a BroadcomNetXtreme Gigabit Ethernet adapter.

I've run the discovery command successfully:

StarNVMeoF_Ctrl.exe discovery_tcp 192.168.11.204:4420 192.168.11.33

StarNVMeoF Controller Application v 2.0.0.672
Our Miniport #4 (ver 2.0 build 672)
Discovering of TCP addr 192.168.11.204:4420 from 192.168.11.33...
NET_LUID: 0x6008015000000
Discovery returned: gen 9, numrec 2
1) subnqn nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery
Discovery: port 0x2, cntlid 0xffff, trtype TCP, addr 192.168.11.205 : 8009, adrfam 1, asqsz 32
2) subnqn nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery
Discovery: port 0x1, cntlid 0xffff, trtype TCP, addr 192.168.11.204 : 8009, adrfam 1, asqsz 32

So now I'm trying to insert_all_tcp and getting the same error message:

StarNVMeoF_Ctrl.exe insert_all_tcp 192.168.11.204:4420 192.168.11.33

StarNVMeoF Controller Application v 2.0.0.672
Our Miniport #4 (ver 2.0 build 672)
Discovering of TCP addr 192.168.11.204:4420 from 192.168.11.33...
NET_LUID: 0x6008015000000
Discovery returned: gen 9, numrec 2
Connecting all TCP devices ...
NET_LUID: 0x6008015000000
Connecting to TCP target at addr 192.168.11.205:8009 from 192.168.11.33 SubNQN=nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery HostNQN=nqn.2008-08.com.starwind.fun nq=0 qd=0 node=0 core=0...
STAR_NVMEOF_IOCTL_TARGET_ADD request failed (err 0, status 0xc00000bb)!
NET_LUID: 0x6008015000000
Connecting to TCP target at addr 192.168.11.204:8009 from 192.168.11.33 SubNQN=nqn.1992-08.com.netapp:sn.0aa8c8d7b26911efbbe5d039ea49c4e5:discovery HostNQN=nqn.2008-08.com.starwind.fun nq=0 qd=0 node=0 core=0...
STAR_NVMEOF_IOCTL_TARGET_ADD request failed (err 0, status 0xc00000bb)!

Maybe this is the problem - When adding the NQN to NetApp, ONTAP9 does not like the formatting that the Starwind Initiator is using and is requesting a :STRING at the end. Is there a way to modify the StarWind Initiator HostNQN?

Screenshot_20250106_NetApp.png (37.74 KiB) Viewed 1060 times

[yaroslav (staff)]

Thanks for your reply. Yes, please try a different NQN for StarWind.
As a side note, make sure that your host is running the latest firmware and drivers.
Also, please make sure MTUs are aligned on both Target and Initiator hosts.

[mark2.tech]

How do I change the NQN on the Starwind Initiator?

EDIT: I changed the host NQN by adding the HostNQN at the end of my insert command:

StarNVMeoF_Ctrl.exe insert_all_tcp 192.168.11.204:4420 192.168.11.33 nqn.2008-08.com.starwind.fun:7767463

I then added this to ONTAP9 and it connected successfully.
Now I need to try this on the VM configuration and will update shortly.
Thank you!

[yaroslav (staff)]

Great!! Thanks for your update. Please keep me posted.

[mark2.tech]

I was able to connect via the Hyper-V switch without SR-IOV setup. I am testing the NVMe performance shortly and will hope to move this to production. Thank you for your help!

[yaroslav (staff)]

You are welcome. Good luck with your project!