iSER_DM.dll causing VSAN to crash

[jamestaylor]

Hi all!

We are testing out setting up a 2 node HA but we are finding that StarWindService.exe crashes sporadically on both nodes (not at the same time) with the following logged in the event log:

Faulting application name: StarWindService.exe, version: 8.0.0.15260, time stamp: 0x64edf8e5
Faulting module name: iSER_DM.dll, version: 8.0.0.15260, time stamp: 0x64edf898
Exception code: 0xc0000005
Fault offset: 0x00000000000016ff
Faulting process id: 0x5b6c
Faulting application start time: 0x01da5dc33997ae97
Faulting application path: C:\Program Files\StarWind Software\StarWind\StarWindService.exe
Faulting module path: C:\Program Files\StarWind Software\StarWind\iSER_DM.dll
Report Id: 171e6fbc-9205-4cf3-b7c3-7584db7eeb64
Faulting package full name:
Faulting package-relative application ID:

We initially had iSER enabled on the sync interfaces (in StarWind.cfg, setting the IPs of the sync interfaces in <iSerListen .../>) and then tried disabling it by commenting out <iSerListen /> but the same issue occurs.

From a previous post on the forum, I think we can rename iSER_DM.dll => iSER_DM.dll.bak to avoid this, however we would like (in the final setup) to use ISER for the sync channels so this is not ideal (happy to try that though if that helps troubleshooting).

Our sync interfaces are using Connect-X3 (not pro) which are working well for SMB Direct, but perhaps are no good for ISER?

ISER does seem to connect when enabled, from what I can make out of the logs, however it's not that quick - I have put that down to the fact that the ConnectX-3s are not (at the moment) directly connected, and are going via an IB switch.

However since disabling ISER does not stop the StarWindService.exe crash, I'm not sure if that is the problem?

We previously setup a 2 node HA using VMs (and no RDMA hardware) which did not crash so... maybe the hardware is to blame?

Info from one of the ConnectX-3s:

Driver Version : 5.50.14740.1
Firmware Version : 2.42.5000
Port Number : 1
Bus Type : PCI-E 8.0 Gbps x8
Link Speed : 40.0 Gbps/Full Duplex
Part Number : MCX353A-FCBT
Device Id : 4099
Revision Id : 0
Current MAC Address : 98-03-9B-DC-C9-91
Permanent MAC Address : 98-03-9B-DC-C9-91
Network Status : Connected
Adapter Friendly Name : Ethernet IB#1
IPv4 Address : 10.254.2.1
Adapter User Name : 0xffff-IPoIB
Adapter PKey : 0xffff (Available)


Many thanks for any assistance!

James

[yaroslav (staff)]

I think there are two issues: iSER support and crashes.
Let us investigate the crashes first. Please share the logs with the timestamps of the crash when iSER was disabled.
Thanks!

[jamestaylor]

Thank you!

I have sent the logs to the support email address - i hope that was correct, much appreciated

[yaroslav (staff)]

Hi,

Please send them together with the timestamps to support@starwind.com use this thread and 1109464 as your references.

[jamestaylor]

Sorry just seen your reply - please ignore my DM - I have emailed them to the correct address as provided now.

Sorry but I don't understand about timestamps - could you explain what I need to provide for timestamps?

Much appreciated!

[anton (staff)]

FYI, we'll continue looking at this issue, but there's few things to know:

(1) Standard way of delivering StarWind VSAN is VM (virtual machine ) now. There's NO performance drawbacks! Windows binaries are considered as a legacy way and have limited functionality.

(2) iSER is deprecated pretty much, it didn't take off on Windows or anywhere else. We're close to release NVMe-oF (RDMA & TCP) StarWind VSAN and THAT would be a killer!

[jamestaylor]

Hi Anton,

Wow I did not realise the CVM approach was an option for me - thats amazing - I have downloaded and setup one HyperV CVM and it all looks perfect so rather than waste your time on the Windows binaries issue, shall I crack on with the VM approach instead? Since as you say that is the future (I have mentioned this to Yaroslav on my reply to the case just now also).

Thanks also for the info regarding iSER - I was hoping to use RDMA for the sync links as its worked really well for us on SMB3 Direct but I can see all of your guides show standard 10gb ethernet iSCSI so it must be fine to use without RDMA. I should be able to use IO-SRV on our ConnectX-3s in ETH mode and get pretty decent iSCSI speeds between the nodes - I will report back!

NVMe-oF... the future is bright

Thanks so much for your help, much appreciated!

[yaroslav (staff)]

You are always welcome. I still would like to look into the service crashes happening after iSER is masked away.

[jamestaylor]

OK great, thank you!

[anton (staff)]

We're phasing out iSER and replacing it with NVMe-oF (both RDMA & TCP) in the next version of StarWind. FYI.

[jamestaylor]

Thanks Anton, looking forward to NVMe-oF, until then we're disabling iSER and using plain iSCSI

[anton (staff)]

That's a good idea

Thanks Anton, looking forward to NVMe-oF, until then we're disabling iSER and using plain iSCSI