Page 1 of 1
NVMe-oF over ROCEv1 possible?
Posted: Wed Mar 03, 2021 1:37 am
by mattaw
Can the StarWind iSCSI Accelerator work with the older ROCEv1 Connectx-3 implementation? (Note: Note Connectx-3 Pro which supports ROCEv2). I am debating whether I need to upgrade to Connectx-3 Pro or better.
[Maybe this should be an additional question, but does the NVMe-oF initiator require Windows 10 Pro for Workstations as a minimum?]
Thanks,
Dr Matthew Swabey
Re: NVMe-oF over ROCEv1 possible?
Posted: Wed Mar 03, 2021 8:42 am
by yaroslav (staff)
Hi,
Welcome to StarWind forum. StarWind iSCSI accelerator load-balances iSCSI sessions (you will need to reconnect those), meaning that it will not utilize ROCE capabilities.
Cheers,
Re: NVMe-oF over ROCEv1 possible?
Posted: Wed Mar 03, 2021 7:31 pm
by mattaw
I am so sorry, yaroslav, that I made a mistake in my previous post and took up your time. I was trying to ask about your NVMe-oF initiator, NOT your iSCSI Accelerator.
So my two questions are:
- Can the NVMe-oF initiator work with the older Connectx-3 ROCEv1
- Does the NVMe-oF require the use of Windows 10 Pro for Workstations or might it work with Windows 10 Pro (Pro Workstation needed to use SMB Direct w. RDMA)
Re: NVMe-oF over ROCEv1 possible?
Posted: Thu Mar 04, 2021 12:19 pm
by yaroslav (staff)
RoCE V1 is still fine.
NVMe-oF Initiator works with Windows 10.
Re: NVMe-oF over ROCEv1 possible?
Posted: Thu Mar 04, 2021 2:24 pm
by mattaw
I really appreciate the time, thank you yaroslav.
Re: NVMe-oF over ROCEv1 possible?
Posted: Thu Mar 04, 2021 2:44 pm
by yaroslav (staff)
You are always welcome
contact us if assistance is required.
Re: NVMe-oF over ROCEv1 possible?
Posted: Fri Mar 05, 2021 5:20 am
by mattaw
After messing about with the undocumented ROCE protocols on the old Connectx-3 EN card, using ROCE V 1.25 on windows (?!? only mentioned occasionally in some Mellanox docs) I was able to get nd_rping / rping running between linux and windows.
I was then able to use Starwind rperf to run RDMA tests which passed.
I was then able to configure and setup a NVMe-oF share of a ZFS zvol on linux -> windows. Performance is out of this world vs. iSCSI. Max read and writes on iSCSI were about 1.9GB/s read and 900MB/s write using ATTO. They are now 2.5GB/s R and W. Not only that but the 4k r/w performance increased by 10x!
Incredible performance. I am going to write a short blog post on it with some benchmarks as soon as I stablize the setup properly between reboots.
Re: NVMe-oF over ROCEv1 possible?
Posted: Fri Mar 05, 2021 8:01 am
by yaroslav (staff)
Awesome! Thanks for sharing your experience.
Re: NVMe-oF over ROCEv1 possible?
Posted: Wed Jan 12, 2022 12:50 pm
by yaroslav (staff)
Small update here. StarWind NMVe-oF Initiator does not work over ROCEv1.