Unable to discover targets using TCP using nvme-of initiator

[courtjesterau]

I have this issue as well

I dont see anywhere you have shared the old build

how do I get the older build?

[yaroslav (staff)]

It was in the support's reply to the ticket you opened with us earlier.

[frollo]

I haven't tried the older build. No chap anywhere, and i don't use RDMA directly as that requires an eterprise license with truenas scale. I can only use via TCP. I will try the older build you shared via the support ticket and report back!

[frollo]

Alright, here is what i found on the old version.

It does in fact work and connect where as the new one does not. I had to completely blow away the share on truenas and i updated the nqn to not have the uuid (which truenas wanted to put by default). After i recreated the share the discovery address which was previously "tcp:nqn.2014-08.org.nvmexpress.discovery" connected directly to the correct one in the ui as "tcp:nqn.2011-06.com.truenas:nvmeof". It connected directly to my old iscsi share without issue and the drive / data is all there.

My entire goal was to replace iscsi with tcp nvmeof for better performance. I haven't really stress tested anything but a quick crystal disk mark shows it is significantly faster than iscsi with way less latency / average response time. This is over a 10G connection.

[yaroslav (staff)]

Thanks for a TrueNAS tip. Good luck with your project.

[bigjme]

I had to completely blow away the share on truenas and i updated the nqn to not have the uuid (which truenas wanted to put by default). After i recreated the share the discovery address which was previously "tcp:nqn.2014-08.org.nvmexpress.discovery" connected directly to the correct one in the ui as "tcp:nqn.2011-06.com.truenas:nvmeof". It connected directly to my old iscsi share without issue and the drive / data is all there.

Just to confirm, the changes you made here were not related to getting the old version able to discover, right?
I'm assuming not but though it best to get clarification for anyone else reading

Also yaroslav, can we get a link to the old version that works for more people to test or do you know if the fixed version is due at some point soon?

Regards,
Jamie

[yaroslav (staff)]

I hope it will be sometime in mid-December.

[yaroslav (staff)]

The link. Should be OK until JAN2.
https://starwind.my.salesforce.com/sfc/ ... 20HsG.mD7Q

[frollo]

I had to completely blow away the share on truenas and i updated the nqn to not have the uuid (which truenas wanted to put by default). After i recreated the share the discovery address which was previously "tcp:nqn.2014-08.org.nvmexpress.discovery" connected directly to the correct one in the ui as "tcp:nqn.2011-06.com.truenas:nvmeof". It connected directly to my old iscsi share without issue and the drive / data is all there.

Just to confirm, the changes you made here were not related to getting the old version able to discover, right?
I'm assuming not but though it best to get clarification for anyone else reading

Also yaroslav, can we get a link to the old version that works for more people to test or do you know if the fixed version is due at some point soon?

Regards,
Jamie

With respect to the changes made. You are correct this was not related to getting the old version to discover. It discovered without issue right away. I had to re-setup the connection so that when i attempted to connect it would handshake correctly and work.

[vinod.k.botla]

We’re evaluating StarWind for using block storage over NVMe-oF on Microsoft Windows server operating systems and facing issues while discovering target storage system (Dell EMC PowerMax). Followed the implementation plan as described. Any help would be much appreciated.

[yaroslav (staff)]

The link. Should be OK until JAN2.
https://starwind.my.salesforce.com/sfc/ ... 20HsG.mD7Q

Try this.

[louis]

I am trying to setup a connection between my TrueNas system and my windows11 Pro PC using TCP as protocol.

It does not work. Using Wireshark I see that the NVME-oF initiator sends a TCP package to the TrueNas server, whci answers with an ACK. For unkown reason the ACK is not accepted and the NVME-oF initiator starts a new session using the same ports. That message is again answered with an ACK and that sequence repeats four times.

Up to the next refresh when it all starts over.

However no real communication and no discovered targets

[yaroslav (staff)]

Can you please attach the Wireshark log?
Make sure
-drivers are up-to-date
-firewalls are opened.
-FW is up-to-date.
-Initiator nqn is allowed on the Target side.

Also, does RDMA connectivity work?

[louis]

Here some info:

I start with the command line version output
TCP
PS C:\WINDOWS\system32> StarNVMeoF_Ctrl.exe discovery_tcp 192.168.18.40:4422 10.5.50.100
StarNVMeoF Controller Application v 2.0.0.702
Our Miniport #4 (ver 2.0.702, lic:1)
Discovering of TCP addr 192.168.18.40:4422 from 10.5.50.100...
NET_LUID: 0x600801A000000
STAR_NVMEOF_IOCTL_DISCOVERY request failed (err 0, status 0xc0000236)!

PS C:\WINDOWS\system32> StarNVMeoF_Ctrl.exe discovery 172.16.10.10:4420 172.16.10.100

RDMA
- I do not think it can work RoCE(v2) is not active on my Connext-4 LX
- e.g. because it is not support on TrueNas community edition which by the way is using the same Connect-4 LX
StarNVMeoF Controller Application v 2.0.0.702
Our Miniport #4 (ver 2.0.702, lic:1)
Discovering of RDMA addr 172.16.10.10:4420 from 172.16.10.100...
STAR_NVMEOF_IOCTL_DISCOVERY request failed (err 0, status 0xc0000034)!

Driver: The windows Connect-4 LX cards are use driver version 3.10.25798.0 which is the latest possible I think
Firewall: pfSense has been opened and I think that is OK given the fact that there is basis TCP-communication between PC and NAS
Firewall: pfSense is running the very latest pFSense+ edition
Network: the elements are interconnected via VLANs one towards the PC which is not aware of that)
and a trunk with another vlan towards TrueNas
The switch is a Mikrotik CRS317
I am using jumbo frames: 9014 for the PC, 9216 for the FW-trunk and the NAS-trunk and 9000 for the pfSense and NAS vlans
Initiator nqn: At this moment all host are allowed
RDMA: does not work as explained above

CaptureSummary.jpg (119.99 KiB) Viewed 30409 times

As requested

Louis
Note I have problems adding a capture I tried to add one inside a zip and one with a jpg extension. Can not drop and place as well get http error. Whatever IO hope it worked

[yaroslav (staff)]

To me, it looks like something is wrong on the networking side.
Please try completely disabling the firewall.
Is there any way to connect the NAS directly to the Initiator host? does ping -f -l 8000 go well? Try setting MTU to 1514. 9014 can be fragmented; 9216 are likely to be fragmented. Yes, it is still often a problem.
Also, are you using a trial or free? Run С:\Program Files\StarWind Software\StarNVMeoF>StarNVMfInitiatorService.exe --checklicense
Please also try this version https://starwind.my.salesforce.com/sfc/ ... 20HsG.mD7Q