Is there a way to force iSER connections? (iSER Troubleshooting)
Posted: Fri Mar 01, 2024 9:25 am
I am currently building my first ever Hyper-V HCI cluster and I found in my testing that iSER sometimes works and sometimes it doesn't. What I mean by that is, if it works (if both nodes show iSER as enabled in the management console), it works but if it doesn't, it is very difficult to get it to work.
What I noticed is that rather than forcing iSER, vSAN seems to use iSER opportunistically. It will try to use iSER but if that fails, it will fall back to non-iSER connections (on the sync interfaces that is).
This opportunistic checking, if iSER should be used, only seems to happen at the start of vSAN. It does not seem to try to activate it later if it didn't work at the start of the service.
Though iSER works sometimes in my test environment, I have encountered the following problems:
This is an example, in this case, a zombie process (starwindservice.exe) is listening on a sync interface (192.168.51.1) per RDMA (netstat -x). Yet another process is listening on the other sync interface (192.168.52.1). despite the StarwindService not running. When I start the service, a new process is spawned which cannot bind to the socket/port. In such a case vSAN logs this error.
By the way, in such a case, I cannot kill the zombie process (starwindservice.exe) without rebooting the computer. Task manager tells me that I do not have the permission to end the process. I ran task manager with administrator privileges.
I wonder,
What I noticed is that rather than forcing iSER, vSAN seems to use iSER opportunistically. It will try to use iSER but if that fails, it will fall back to non-iSER connections (on the sync interfaces that is).
This opportunistic checking, if iSER should be used, only seems to happen at the start of vSAN. It does not seem to try to activate it later if it didn't work at the start of the service.
Though iSER works sometimes in my test environment, I have encountered the following problems:
- iSER is not used at all. All sync interfaces show iSER as disabled in the management console.
- iSER only works in one direction. My setup has two nodes and sometimes one node (say: node 1 for example) does have an iSER connection to node 2 but node 2 only has a non-iSER connection to node 1.
Code: Select all
>netstat -xan | select-string 'PID|3260'
Mode IfIndex Type Local Address Foreign Address PID
User 18 Listener 192.168.51.1:3260 NA 2964
User 9 Listener 192.168.52.1:3260 NA 2965
Code: Select all
2/28 21:04:53.451695 d90 iSER: *** iSerDmServerSocket::Listen: IND2Listener::Bind failed with c0000043
2/28 21:04:53.451715 d90 Srv: *** iScsiServer::listenConnections: iSER listen to 192.168.52.1:3260 failed with 3221225539 (0xc0000043).
I wonder,
- Is there a way to force iSER connections? That is, to tell vSAN to not fall back to a regular iSCSI connection on the sync interface if an iSER connection cannot be established right now?
- Will vSAN opportunistically try to establish an iSER connection continuously, if that failed during startup, and a regular iSCSI connection was used for syncing?
- Is there a way to kill the zombie process without rebooting (if the vSAN service is already stopped).
- Is it on purpose that the starwindservice.exe process is protected from being killed?
- What could be the reason why the zombie process is left behind anyway, if the service is stopped?
I tried using both of these methods:andCode: Select all
get-service StarwindService | restart-service
In both cases, sometimes it doesn't seem to properly shut down and end the process.Code: Select all
& 'C:\Program Files\StarWind Software\StarWind\ServiceLaunchMgr.exe' /servicename:starwindservice /operation:restart