Unable to discover targets using TCP using nvme-of initiator

[louis]

The firewall did not allow pings into the greenzone but I changed that for my personal PC-lan. I allow the firewall no to pass IPV4 all ports and protocols.

Here the ping results
PS C:\WINDOWS\system32> ping -f -l 8000 192.168.18.40

Pinging 192.168.18.40 with 8000 bytes of data:
Reply from 192.168.18.40: bytes=8000 time<1ms TTL=63
Reply from 192.168.18.40: bytes=8000 time<1ms TTL=63
Reply from 192.168.18.40: bytes=8000 time<1ms TTL=63
Reply from 192.168.18.40: bytes=8000 time<1ms TTL=63

I can temporarily place my pc in the truenas vlan, but that is more work. I have to reconfigure the switch. I do not think that would solve the problem. What ever if I do that tomorrow.

Changing the frame sizes is another thing that would affect far more. I don't think I am going to do that. Jumbo frames help me to get faster transfers for info I actually get almost 10G via my 10G network for SMB and 5G up and 10G down when using iSCSI All via the same route and the same firewall as described.

Note: max frame size of the trunk (and switches) should always be a bit bigger than the frame size of the vlan's IMHO (a size bigger than the trunk fame is of course not accepted)

[yaroslav (staff)]

Please try an older version and simplify the networking as much as you can.

[louis]

If you look what happens you see that:

- the nvme-initiator sends a package
- and it receives an ack

Unless the ack is somehow not ok, that is exactly what should happen!
So the communication via the network on tcp level between the NAS and the PC is OK !!!

What seems to be wrong is that the nvme-initiator does not recognise the ack-frame as ack!
That is a a problem in the nvme-initiator !
Not in the network !!

Now I can of course try to change the network in such a way that the frame exactly is what works for the nvme-initiator,
however that is working around the problem, in state of fixing the problem!

So unless I am wrong (check the traces), I think I should leave my network as it is, including jumbo frames.

[yaroslav (staff)]

I need to simplify the setup, namely, the network stack by using direct link, no teaming (if applicable) and no firewall to be able to see that it is initiator for sure.
Please also share the output for the license file I asked earlier and downgrade the initiator.

[louis]

As requested here a copy of the licence file screen shot.

As requested I will try with the older version 'StarWind NVMe-oF Initiator.1.9.0.596(rev 598).Setup.462.exe'

But I think I am not going to change my network, since the network is behaving as it shoot IMHO. And perhaps even more important, if I change the network and suppose it works, that only tells that the NVME-oF Initiator can handle that simple network ...... not that the NVME-Initiator does not have a bug !!

[louis]

I did uninstall the actual version and did install the old version.

Some remarks:
- after removing the old version the 'C:\ProgramData\StarWind' directory was not removed. I did rename that directory before the new install
- the driver is installed and working (computer management shows a working the driver after reboot)
- after installing the indicated old version, I noticed there was no gui So I tried using the command line with the result show below
- note there was no licence request in the old version

C:\Program Files\StarNVMeoF>StarNVMeoF_Ctrl.exe discovery 192.168.18.40:4422 10.5.50.100

StarNVMeoF Controller Application v 1.9.0.596
Our Miniport #4 (ver 1.9 build 596)
Discovering of RDMA addr 192.168.18.40:4422 from 10.5.50.100...
NET_LUID: 0x600801A000000
STAR_NVMEOF_IOCTL_DISCOVERY request failed (err 121)!
Discovery() failed!

It does not seems to work

PS:
- adding attachments to the post ..... does not work as it shoot
- I do have yesterday capture files an log files which I could upload preferable private via email

[bigjme]

Ok let me just post a bit of information here and just point out a few things to prevent people going around in circles and arguing over the same things

Truenas
Only TCP is available or discoverable on consumer hardware regardless of what you have in the server or initiator unless you have an enterprise license.
RDMA will not work in the default Truenas NVMe-OF instance unless you have an enterprise license as they do not make the RDMA endpoints discoverable.

@louis Use the windows iscsi gui and select the StarWinds Initiator when you have the older version.
It does work and i have it confirmed working on my systems using Connect X4's, but the newer version with the Gui only works for RDMA which you wont be able to use

StarWind
RDMA works fine on both the current and old versions of the initiator be it a SPDK or Truenas (nvmet) server
TCP does not discover either of these confirmed working servers in the new version, but does in the older version you have linked

I have confirmed this to be the case on both newer and older Truenas installs as i have custom NVMe-OF setups that are again, confirmed to work with no network issues

I hope this helps stop some of the arguments/disagreements

[louis]

Oeps! TCP does not discover either of these confirmed working servers in the new version, but does in the older version you have linked.
That is not what the software and the GUI suggests !

Oeps! It does work and i have it confirmed working on my systems using Connect X4's, but the newer version with the Gui only works for RDMA which you wont be able to use

Use the windows iscsi gui and select the StarWinds Initiator when you have the older version.
@bigjme Could you explain how to do that?

If I start the ^Windows iSCSI-initiator^ (where I have the normal iSCSI-drives), I can add the target portal like it is a normal iSCSI-portal ,,,,
However I have no idea how to 'connect' to the 'starlink Initiator' from there !

As shown in the previous post I did also try via the command line I tried that again now with the _tcp option. Did not work.

[bigjme]

Under the windows Iscsi Initiator with the old version installed:
Discovery -> Discovery Portal -> Advanced -> Local adaptor: Starwind -> Initiator ip: Which ever is right -> Set the ip for the Truenas server -> Port set to usually 4420 to match Truenas

You should see entries like this show up after discovery:

[yaroslav (staff)]

Just a small remark on the GUI: Discovery, both CLI and UI, might not work if the trial License has expired.
Free does not have UI, though.

suppose it works, that only tells that the NVME-oF Initiator can handle that simple network ...... not that the NVME-Initiator does not have a bug !!

This is not what I am saying. I am dealing with a "sandwich" now. I want to peel off the layers. To get to the bottom of it. Please cooperate.

[yaroslav (staff)]

@bigjme,

Thanks for sharing your experience.

[louis]

This is not what I am saying. I am dealing with a "sandwich" now. I want to peel off the layers. To get to the bottom of it. Please cooperate.

yaroslav (staff)@ Sorry, but I try to cooperate as much as possible! And I did put a lot of work in that! I still do!

I did follow bigjme@ its advice after switching to the old version however, the old version does not work as well. Even worse I do not even see any TCP towards the NAS (what I do see with the new version).

I think we should try to solve the issue with the new version. The new version should also support TCP (which it does according to the GUI) and that is the version being supported not the old one I suppose.

Related to fragmentation
I did further test with the earlier suggested ping. In the Wireshark traces there is a line saying packages should be 8960. So I did test that.
C:\Users\Louis>ping -f -l 8960 192.168.18.40 that is OK with my jumbo frame setting from 9014 (I think that is the suggested default)
The max for the Connect-X4 jumbo 9014 using windows without (local) vlan turned out to be 8072
C:\Users\Louis>ping -f -l 8974 192.168.18.40

Pinging 192.168.18.40 with 8974 bytes of data:
Packet needs to be fragmented but DF set

Related to using the old version
What IMHO is a bad idea, unless the new version really does not support TCP !!
Some pictures below, as said it does NOT work ! Not even one TCP towards the NAS seen!
(windows 11 pro latest version 2025 H2).

Below should be the picture .... however adding files .... however sorry I do not manage ... html issue ??
I hope the picture is attached.

[louis]

On special request I did perform tests with the PC and the NAS in the same VLAN.

I performed the test with the old StarWindDriver => does not work and I did not see TCP-packages

The I removed the old StarWindDriver => no TCP-packages. However after rebooting th PC there were again TCP-packages.
Unfortunately exactly the same behavour as before
- driver sends package
- NAS sends ACK
- driver does not recognize and resend the package

In the end exactly the same behavoir as before and of course no recognized targets.

Note that I also repeated the ping test "ping -f -l 8960 192.168.18.40" which went OK

All completely conform (my) expectations regretfully.

[yaroslav (staff)]

Please share the checklicense output.

[louis]

Sorry I did try to share a screen copy, but I do not manage to add attachments

In one of my first post I did manage to add a picture, but perhaps due to the fact that I update my WIndows11 system to 25H2, whatever I try I did not manage to add attachments to my posts! An error is displayed Error HTTP-error Among them that picture and wireshark traces.
Note that if I open the license window in the GUI the license is shown. I did attache the screen capture again, but I do think it does not work!

You ask for the licence info and I am not sure if you refer to the described screen or a cmd output. So I tried ^StarNVMeoF_Ctrl.exe checklicense^ which did not work.

Next to this issue let me describe my yesterday actions in more detail:

PC to NAS-vlan
I did make changes to my 10G switch so that my PC became part op the NAS-vlan

Using the old driver
- did remove the new driver. Note that the C:\ProgramData\StarWind folder was not removed perhaps intentionally, however not ok, I renamed it
- did install the old driver and tested it using the iSCSI-gui
- was not necessary to install the license
- no C:\ProgramData\StarWind folder created
- did not work not; any TCP messages; no target of course
- removed the old driver
- driver still present in the driver management screen
- after removing the driver windows boot did take an aw full lot of time
- driver gone after reboot
- checking the iSCSI-controller the nvme-driver was gone and replaced by ''rootscsi.controller"
Not OK I removed it.
- I did not reboot after the installation of the old driver, given my actual experience with the new driver
and earlier experiences I should perhaps have tried that

Installing the new driver
- old driver version did not work, and using that one is also probably not a good idea !!
- installing the new driver
- note that i did not need to install the license, was all ready there from earlier installs
- ping -f -l 8960 192.168.18.40 OK (tested with 8960 because that is the packet size according to wireshark traces)
- no TCP-messages at all
- reboot the system, just to try (should not necessary), window storage controller was installed and active
- ^surprise^ driver does send a package to the NAS now
- NAS sends ACK back; ACK not seen or accepted by the driver
- driver resend the TCP-package
- NAS sends ACK all just as before
- etc
- So no real communication and of course no targets found