Long story short, i got it turn on now but facing errors.
iSCSI works well when switching to RoCE it fails with the following errors.
The esxi server and windows nas , both are using mellanox connectx-3 interface, it should supports ROCE v1 (and v1.5) only.
When set to RoCE v1 using Set-MlnxDriverCoreSetting -RoceMode 1
From ESXi vmkernel.log
12/10 13:16:43.386 13c0 iSER: iSerDmSocket::InitQueues: Created QueuePair (recv 264, init 1056, cq 1320, group 0, affinity 0x3).
12/10 13:16:43.386 13c0 iSER: iSerDmSocket::Accept: iSER: Accept: using Ird = 0, Ord = 16.
12/10 13:16:43.386 13c0 iSER: *** iSerDmSocket::Accept: IND2Connector::Accept failed with c000009a
12/10 13:16:43.386 13c0 iSER: *** iSerDmServerSocket::Accept: Socket::Accept failed with c000009a
12/10 13:16:43.386 13c0 Srv: iScsiServer::listenConnections: Accepted iSER connection from 10.10.10.1:41688 to 10.10.10.20:3260. (Id = 0x1)
12/10 13:16:43.386 13c0 S[1]: iScsiSession::iScsiSession: Session (00000224D752E0C0)
12/10 13:16:43.386 13c0 C[1], FREE: iScsiConnection::doTransition: Event - CONNECTED.
12/10 13:16:43.386 13c0 C[1], XPT_UP: iScsiConnection::fsmT3: T3.
12/10 13:16:43.386 13c0 iSER: iSerDmSocket::InitQueues: Created QueuePair (recv 264, init 1056, cq 1320, group 0, affinity 0x3).
12/10 13:16:43.386 13c0 iSER: iSerDmSocket::Accept: iSER: Accept: using Ird = 0, Ord = 16.
12/10 13:16:43.386 13c0 iSER: *** iSerDmSocket::Accept: IND2Connector::Accept failed with c000009a
12/10 13:16:43.386 13c0 iSER: *** iSerDmServerSocket::Accept: Socket::Accept failed with c000009a
12/10 13:16:43.386 13c0 Srv: iScsiServer::listenConnections: Accepted iSER connection from 10.10.10.1:41689 to 10.10.10.20:3260. (Id = 0x2)
12/10 13:16:43.386 13c0 S[2]: iScsiSession::iScsiSession: Session (00000224D72B0FC0)
12/10 13:16:43.386 13c0 C[2], FREE: iScsiConnection::doTransition: Event - CONNECTED.
12/10 13:16:43.386 13c0 C[2], XPT_UP: iScsiConnection::fsmT3: T3.
2018-12-10T05:16:43.396Z cpu16:2098053)iser: iser_EPConnect: src ipv4 10.10.10.1
2018-12-10T05:16:43.396Z cpu16:2098053)iser: iser_EPConnect: dst ipv4 10.10.10.20
2018-12-10T05:16:43.396Z cpu16:2098053)WARNING: rdmaDriver: RDMAGetValidGidType Protocol not supported by device
2018-12-10T05:16:43.396Z cpu16:2098053)WARNING: rdmaDriver: RDMACM_BindLegacy Underlying device does not support requested gid/RoCE type. Failed with status: Protocol not supported
2018-12-10T05:16:43.397Z cpu9:2097736)iser: iser_RDMACMEventHandler: iser conn 0x430b457dfce0 cm event 0
2018-12-10T05:16:43.397Z cpu16:2098053)iser: iser_EPConnect: src ipv4 10.10.10.1
2018-12-10T05:16:43.397Z cpu16:2098053)iser: iser_EPConnect: dst ipv4 10.10.10.20
2018-12-10T05:16:43.397Z cpu16:2098053)WARNING: rdmaDriver: RDMAGetValidGidType Protocol not supported by device
2018-12-10T05:16:43.397Z cpu16:2098053)WARNING: rdmaDriver: RDMACM_BindLegacy Underlying device does not support requested gid/RoCE type. Failed with status: Protocol not supported
2018-12-10T05:16:43.398Z cpu16:2098053)Binding for id: 0x430b457e26a0 exists, increment refcount
2018-12-10T05:16:43.398Z cpu16:2098053)WARNING: rdmaDriver: RDMACM_BindLegacy RDMA CM Binding already exists.
2018-12-10T05:16:43.398Z cpu19:2097784)iser: iser_RDMACMEventHandler: iser conn 0x430b457dfce0 cm event 2
2018-12-10T05:16:43.398Z cpu9:2097736)iser: iser_RDMACMEventHandler: iser conn 0x430b457e13c0 cm event 0
2018-12-10T05:16:43.399Z cpu19:2097784)iser: iser_ComplSlabInit: min 0 max 41048
2018-12-10T05:16:43.399Z cpu19:2097784)iser: iser_RouteResolvedHandler: iser conn: 0x430b457dfce0 uplink: vmnic4
2018-12-10T05:16:43.399Z cpu19:2097784)iser: iser_RDMACMEventHandler: iser conn 0x430b457e13c0 cm event 2
2018-12-10T05:16:43.399Z cpu21:2104896)iser: iser_DisconnectWorld: 0x430b457dfce0 iser conn disconnect world init
2018-12-10T05:16:43.400Z cpu19:2097784)iser: iser_ComplSlabInit: min 0 max 41048
2018-12-10T05:16:43.400Z cpu19:2097784)iser: iser_RouteResolvedHandler: iser conn: 0x430b457e13c0 uplink: vmnic4
2018-12-10T05:16:43.400Z cpu6:2104897)iser: iser_DisconnectWorld: 0x430b457e13c0 iser conn disconnect world init
2018-12-10T05:16:43.649Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:43.652Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:43.908Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:43.911Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.398Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.401Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.657Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.660Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.915Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:44.918Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.399Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.402Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.658Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.661Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.914Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:45.917Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.403Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.405Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.660Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.664Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.919Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:46.922Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.405Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.408Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.663Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.667Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.922Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:47.925Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.408Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.411Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.666Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.669Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.924Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:48.927Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:49.410Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457dfce0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:49.413Z cpu16:2098053)iser: iser_EPPoll: Polling ep/iser_conn 0x430b457e13c0 transport 0x430b457dfa00 timeoutMS 1
2018-12-10T05:16:49.668Z cpu16:2098053)iser: iser_EPDisconnect: transport 0x430b457dfa00 ep 0x430b457dfce0
2018-12-10T05:16:49.668Z cpu16:2098053)iser: iser_EPDisconnect: Waiting for QP to transition to error state ...
2018-12-10T05:16:49.678Z cpu16:2098053)iser: iser_FreeIbConnRes: freeing iser_conn 0x430b457dfce0 ib resources
2018-12-10T05:16:49.734Z cpu21:2104896)iser: iser_DisconnectWorld: 0x430b457dfce0 iser conn disconnect world wake up: World is marked for death
2018-12-10T05:16:49.734Z cpu21:2104896)iser: iser_DisconnectWorld: 0x430b457dfce0 disconnect world wokeup
2018-12-10T05:16:49.736Z cpu16:2098053)iser: iser_EPDisconnect: transport 0x430b457dfa00 ep 0x430b457e13c0
2018-12-10T05:16:49.736Z cpu16:2098053)iser: iser_EPDisconnect: Waiting for QP to transition to error state ...
2018-12-10T05:16:49.746Z cpu16:2098053)iser: iser_FreeIbConnRes: freeing iser_conn 0x430b457e13c0 ib resources
2018-12-10T05:16:49.804Z cpu6:2104897)iser: iser_DisconnectWorld: 0x430b457e13c0 iser conn disconnect world wake up: World is marked for death
2018-12-10T05:16:49.804Z cpu6:2104897)iser: iser_DisconnectWorld: 0x430b457e13c0 disconnect world wokeup