NVMe-oF over ROCEv1 possible?

[mattaw]

Can the StarWind iSCSI Accelerator work with the older ROCEv1 Connectx-3 implementation? (Note: Note Connectx-3 Pro which supports ROCEv2). I am debating whether I need to upgrade to Connectx-3 Pro or better.

[Maybe this should be an additional question, but does the NVMe-oF initiator require Windows 10 Pro for Workstations as a minimum?]

Thanks,

Dr Matthew Swabey

[yaroslav (staff)]

Hi,

Welcome to StarWind forum. StarWind iSCSI accelerator load-balances iSCSI sessions (you will need to reconnect those), meaning that it will not utilize ROCE capabilities.

Cheers,

[mattaw]

I am so sorry, yaroslav, that I made a mistake in my previous post and took up your time. I was trying to ask about your NVMe-oF initiator, NOT your iSCSI Accelerator.

So my two questions are:

1. Can the NVMe-oF initiator work with the older Connectx-3 ROCEv1
2. Does the NVMe-oF require the use of Windows 10 Pro for Workstations or might it work with Windows 10 Pro (Pro Workstation needed to use SMB Direct w. RDMA)

[yaroslav (staff)]

RoCE V1 is still fine.
NVMe-oF Initiator works with Windows 10.

[mattaw]

I really appreciate the time, thank you yaroslav.

[yaroslav (staff)]

You are always welcome contact us if assistance is required.

[mattaw]

After messing about with the undocumented ROCE protocols on the old Connectx-3 EN card, using ROCE V 1.25 on windows (?!? only mentioned occasionally in some Mellanox docs) I was able to get nd_rping / rping running between linux and windows.

I was then able to use Starwind rperf to run RDMA tests which passed.

I was then able to configure and setup a NVMe-oF share of a ZFS zvol on linux -> windows. Performance is out of this world vs. iSCSI. Max read and writes on iSCSI were about 1.9GB/s read and 900MB/s write using ATTO. They are now 2.5GB/s R and W. Not only that but the 4k r/w performance increased by 10x!

Incredible performance. I am going to write a short blog post on it with some benchmarks as soon as I stablize the setup properly between reboots.

[yaroslav (staff)]

Awesome! Thanks for sharing your experience.

[yaroslav (staff)]

Small update here. StarWind NMVe-oF Initiator does not work over ROCEv1.

[ichundes]

FWIW the initiator works fine for me with ROCEv1 on ConnectX-3 (Non-Pro) cards between a Windows initiator and Ubuntu 24.04 target.

[yaroslav (staff)]

Many thanks for testing and welcome to StarWind Forum.