Native Infiniband

peksi · Sun Apr 12, 2020 12:58 pm

Hi. I am looking for a new solution to replace our aging storage system. We use storage system mainly to provide block storage to KVM/Libvirt. We are also considering VMware as a virtualization platform so the storage system should provide us with block storage and possible VMware compatible storage.

If I understood correctly the most cost efficient way to accomplish high performance storage network is native Infiniband. Compared to Ethernet solutions I can save thousands and still be able to do 56GB connections with less than millisecond latency. If I were to go Ethernet way (RDMA with ROCE/iSer) I would need to have lossless ethernet and pay thousands if not tens of thousands for appropriate switches. Did i get it right or is there more cost efficient way to do RDMA over ethernet?

Which brings me to my question. Can StarWind VSAN do native 56 gigabit infiniband storage networking? I mean infiniband without IP. I wish to access storage NVME target using low latency 56G Infiniband without Ethernet overhead.

Sorry for incorrect terminology, hope you get my thought. Thanks in advance.

Mon Apr 13, 2020 7:59 am

Hello,

To start with, are we discussing a converged (i.e., storage and server that hosts VMs in different boxes) or hyperconverged (i.e., storage is in the box that hosts VMs) setup?

In general, RDMA is a great step towards building a modern environment. I'd say it may sometimes be the must for IOPS-hungry systems.
However, going RDMA makes sense only if you have powerful underlying storage and CPU utilization issues.
May I learn more about the underlying storage you have? If that is all-NVMe storage, try StarWind's implementation of NVMe-oF (works over RoCE, not sure if that works for you). We created both Target and Initiator. Please note that NVMe-oF is still an experimental feature so think good before bringing it to production.

It is also possible to tweak system performance to ensure high performance. I am talking about system and StarWind VSAN settings.

I mean infiniband without IP.

StarWind VSAN won't work like that. It needs IP and we need Ethernet. May I learn which exactly NICs and switches you plan to use?

As for VMware, I am not entirely sure if ESXi works properly with PCIe disks. Yes, you can use an NVMe controller for VMs, but I have never tried that thing to be honest.
Furthermore, I have something to say about iSER (VSAN also supports that thing as an experimental feature). ESXi may not work correctly with it. I'd say that is a matter of drivers...
You can either try using KVM or Windows as we did to set a performance record. Heard that CERN have their cluster running on KVM

Useful links:
How to create NVMe-oF Target: https://www.starwindsoftware.com/resour ... g-nvme-of/
How to tune NVMe-oF Initiator: https://www.starwindsoftware.com/resour ... initiator/
Performance record with our Implementation of NVMe-oF (it was a Windows setup): https://www.starwindsoftware.com/hyperc ... high-score and https://www.starwindsoftware.com/nvme-o ... me-cluster.

Let me know if there are any additional questions.

peksi · Tue Apr 14, 2020 9:24 am

Thanks for your input. All advice and suggestions are welcome as all options are open now.

We are talking about converged setup. Right now we're using KVM virtualization based on iSCSI block devices, which are exported by a proprietary storage server. We have total of 20 guests, including self hosted ISP and cloud services and we also have a growing number of development and virtual desktop environments for multiple customers.

Our current solution is too slow due to storage system limitations. Reliability is of concern too, although current solution has been working stably.

I wish to procure a storage system that is future proof and as price efficient as possible. RDMA would be preferred and it would be price-wise accessible to us via Infiniband. Ethernet solutions are too expensice due to PFC switches (if I understood correctly?). We have 2 existing Mellanox 56G IB switches and would like to utilize them.

I was originally considering SAS3 SSD based solution but U.2 SSD is not that much more expensive and would give me so much more performance.

ESXi is just another card on the table which I am not that familiar with but what would seem like a stable system. We can continue with KVM without a sweat as well.

What kind of virtual + storage system would you recommend in this case?

Tue Apr 14, 2020 1:00 pm

We have 2 existing Mellanox 56G IB switches and would like to utilize them.

That's a good idea. Unfortunately, StarWind VSAN does not work over InfiniBand yet.

ESXi is just another card on the table which I am not that familiar with but what would seem like a stable system. We can continue with KVM without a sweat as well.

KVM would be just good because IDK if ESXi works well with NVMe-like stuff.

What kind of virtual + storage system would you recommend in this case?

StarWind VSAN inside a Windows Server VM works over Ethernet. We can work over RoCE (VSAN has NVMe-oF Target working in experimental mode) or iSER (experimental feature too). StarWind VSAN for vSphere (one coming inside a CentOS VM) does not have those features, i.e., it works only over iSCSI.
You can always try POC and see if you can get the performance you are up for. However, one more time, StarWind VSAN still needs Ethernet.