NVMe-oF over ROCEv1 possible?

Initiator (iSCSI, FCoE, AoE, iSER and NVMe over Fabrics), iSCSI accelerator and RAM disk

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
mattaw
Posts: 7
Joined: Tue Mar 02, 2021 6:13 pm

Wed Mar 03, 2021 1:37 am

Can the StarWind iSCSI Accelerator work with the older ROCEv1 Connectx-3 implementation? (Note: Note Connectx-3 Pro which supports ROCEv2). I am debating whether I need to upgrade to Connectx-3 Pro or better.

[Maybe this should be an additional question, but does the NVMe-oF initiator require Windows 10 Pro for Workstations as a minimum?]

Thanks,

Dr Matthew Swabey
yaroslav (staff)
Staff
Posts: 2341
Joined: Mon Nov 18, 2019 11:11 am

Wed Mar 03, 2021 8:42 am

Hi,

Welcome to StarWind forum. StarWind iSCSI accelerator load-balances iSCSI sessions (you will need to reconnect those), meaning that it will not utilize ROCE capabilities.

Cheers,
mattaw
Posts: 7
Joined: Tue Mar 02, 2021 6:13 pm

Wed Mar 03, 2021 7:31 pm

I am so sorry, yaroslav, that I made a mistake in my previous post and took up your time. I was trying to ask about your NVMe-oF initiator, NOT your iSCSI Accelerator.

So my two questions are:
  1. Can the NVMe-oF initiator work with the older Connectx-3 ROCEv1
  2. Does the NVMe-oF require the use of Windows 10 Pro for Workstations or might it work with Windows 10 Pro (Pro Workstation needed to use SMB Direct w. RDMA)
yaroslav (staff)
Staff
Posts: 2341
Joined: Mon Nov 18, 2019 11:11 am

Thu Mar 04, 2021 12:19 pm

RoCE V1 is still fine.
NVMe-oF Initiator works with Windows 10.
mattaw
Posts: 7
Joined: Tue Mar 02, 2021 6:13 pm

Thu Mar 04, 2021 2:24 pm

I really appreciate the time, thank you yaroslav.
yaroslav (staff)
Staff
Posts: 2341
Joined: Mon Nov 18, 2019 11:11 am

Thu Mar 04, 2021 2:44 pm

You are always welcome :) contact us if assistance is required.
mattaw
Posts: 7
Joined: Tue Mar 02, 2021 6:13 pm

Fri Mar 05, 2021 5:20 am

After messing about with the undocumented ROCE protocols on the old Connectx-3 EN card, using ROCE V 1.25 on windows (?!? only mentioned occasionally in some Mellanox docs) I was able to get nd_rping / rping running between linux and windows.

I was then able to use Starwind rperf to run RDMA tests which passed.

I was then able to configure and setup a NVMe-oF share of a ZFS zvol on linux -> windows. Performance is out of this world vs. iSCSI. Max read and writes on iSCSI were about 1.9GB/s read and 900MB/s write using ATTO. They are now 2.5GB/s R and W. Not only that but the 4k r/w performance increased by 10x!

Incredible performance. I am going to write a short blog post on it with some benchmarks as soon as I stablize the setup properly between reboots.
yaroslav (staff)
Staff
Posts: 2341
Joined: Mon Nov 18, 2019 11:11 am

Fri Mar 05, 2021 8:01 am

Awesome! Thanks for sharing your experience.
yaroslav (staff)
Staff
Posts: 2341
Joined: Mon Nov 18, 2019 11:11 am

Wed Jan 12, 2022 12:50 pm

Small update here. StarWind NMVe-oF Initiator does not work over ROCEv1.
Post Reply