iSER support?

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
mserra
Posts: 4
Joined: Fri Nov 28, 2014 11:38 am

Fri Nov 28, 2014 12:44 pm

Hello!

in regards of using Mellanox ConnectX-3 Pro cards along with Starwind iSCSI software, I was wondering whether we could use iSER and make use of RDMA. Do you know if that would be possible? Any tips/suggestions? I'm actually looking forward to Anton's experience with Mellanox on his blog.

thank you,

Marc Serra
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Fri Nov 28, 2014 4:57 pm

If you mean StarPort then you're not going to see any support for iSER anytime soon... If you mean StarWind Virtual SAN then the problem is we can have it many times implemented @ our side but MSFT is not going to support it with their initiator making our primary production scenario being out of the game. So we optimize our internal protocol (iSCSI-based) for synchronization and allow MSFT to push SMB3 where SMB Direct (equivalent of an iSER) is present. For our Compute and Storage Separated scenario (StarWind runs block back bone for MSFT Scale-Out File Server).
mserra wrote:Hello!

in regards of using Mellanox ConnectX-3 Pro cards along with Starwind iSCSI software, I was wondering whether we could use iSER and make use of RDMA. Do you know if that would be possible? Any tips/suggestions? I'm actually looking forward to Anton's experience with Mellanox on his blog.

thank you,

Marc Serra
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
bacterialbag
Posts: 14
Joined: Wed Nov 19, 2014 12:57 am

Fri Nov 28, 2014 11:36 pm

Could you clarify this for me?

I dont understand a word what you're saying.

Im using Mellanox Connectx3-Pro's myself, and was wondering if I could use any of the offloading/RDMA/Infiniband capabilities with starwind? Or will my cards only be useable with IPoIB and all the extras go to waste?

Please help clarify what infiniband options I can and can not use.

Thanks,

- Bag
anton (staff) wrote:If you mean StarPort then you're not going to see any support for iSER anytime soon... If you mean StarWind Virtual SAN then the problem is we can have it many times implemented @ our side but MSFT is not going to support it with their initiator making our primary production scenario being out of the game. So we optimize our internal protocol (iSCSI-based) for synchronization and allow MSFT to push SMB3 where SMB Direct (equivalent of an iSER) is present. For our Compute and Storage Separated scenario (StarWind runs block back bone for MSFT Scale-Out File Server).
mserra wrote:Hello!

in regards of using Mellanox ConnectX-3 Pro cards along with Starwind iSCSI software, I was wondering whether we could use iSER and make use of RDMA. Do you know if that would be possible? Any tips/suggestions? I'm actually looking forward to Anton's experience with Mellanox on his blog.

thank you,

Marc Serra
bacterialbag
Posts: 14
Joined: Wed Nov 19, 2014 12:57 am

Fri Nov 28, 2014 11:37 pm

Could you clarify this for me?

I dont understand a word what you're saying.

Im using Mellanox Connectx3-Pro's myself, and was wondering if I could use any of the offloading/RDMA/Infiniband capabilities with starwind? Or will my cards only be useable with IPoIB and all the extras go to waste?

Please help clarify what infiniband options I can and can not use.

Thanks,

- Bag
anton (staff) wrote:If you mean StarPort then you're not going to see any support for iSER anytime soon... If you mean StarWind Virtual SAN then the problem is we can have it many times implemented @ our side but MSFT is not going to support it with their initiator making our primary production scenario being out of the game. So we optimize our internal protocol (iSCSI-based) for synchronization and allow MSFT to push SMB3 where SMB Direct (equivalent of an iSER) is present. For our Compute and Storage Separated scenario (StarWind runs block back bone for MSFT Scale-Out File Server).
mserra wrote:Hello!

in regards of using Mellanox ConnectX-3 Pro cards along with Starwind iSCSI software, I was wondering whether we could use iSER and make use of RDMA. Do you know if that would be possible? Any tips/suggestions? I'm actually looking forward to Anton's experience with Mellanox on his blog.

thank you,

Marc Serra
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Sat Nov 29, 2014 12:30 pm

It completely depends on the way you're going to use them.

1) You're using "Compute and Storage Separated" scenario you'll use RDMA/IB together with SMB3 protocol on top of it with Hyper-V nodes "talking" to Scale-Out File Server setup. StarWind will run @ SoFS nodes and do [own nameless protocol]-over-iSCSI in the background to keep CSVs on SoFS nodes in synch between each other (similar to upcoming Microsoft Storage Spaces Shared Nothing but w/o "slabs", multiple re-routes and not with 4 nodes minimum, but that's other story...). StarWind does not expose iSCSI to Hyper-V with this setup.

2) You're using "Hyper-Converged" scenario then you'll have to switch IB -> IP-over-IB (we get ±33-35 Gbps on a 56 Gb IB now in the lab, ping Bohdan for more details if you want). You're going to have offload/RDMA and so on but inside: there's no **native** support for IB or RDMA with StarWind iSCSI at this moment. From the other point it's not going to help much for a reason: talking on the same node Hyper-V -to- Hyper-V we do DMA with an own driver stack and not RDMA as TCP and iSCSI stacks are mostly bypassed and with synchronization there's own protocol over iSCSI (see above) and it uses huge blocks to move data other Etherent so RDMA latency does not really matter, raw bandwidth is fine.
bacterialbag wrote:Could you clarify this for me?

I dont understand a word what you're saying.

Im using Mellanox Connectx3-Pro's myself, and was wondering if I could use any of the offloading/RDMA/Infiniband capabilities with starwind? Or will my cards only be useable with IPoIB and all the extras go to waste?

Please help clarify what infiniband options I can and can not use.

Thanks,

- Bag
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
mserra
Posts: 4
Joined: Fri Nov 28, 2014 11:38 am

Mon Dec 01, 2014 2:49 pm

I'm actually planning on going with Mellanox ConnectX-3 Pro + Starwind to serve as SAN for our VMWare Infraestructure (which will also use ConnectX-3 Pro). I'm interested in the best approach for maximum speed & reliability and that's why the RDMA question came to mind. I still don't have all the hardware yet but as soon as I have it I intend to try with different setups and see what's the best option. I might ping Bohdan as suggested too :)

Thanks for your replies,

Marc Serra
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Mon Dec 01, 2014 3:39 pm

So it's "Compute and Storage Separated" and you plan to use iSCSI as a connectivity to VMware vSphere. OK :) We'll re-check current status of iSER with vSphere as a client.
mserra wrote:I'm actually planning on going with Mellanox ConnectX-3 Pro + Starwind to serve as SAN for our VMWare Infraestructure (which will also use ConnectX-3 Pro). I'm interested in the best approach for maximum speed & reliability and that's why the RDMA question came to mind. I still don't have all the hardware yet but as soon as I have it I intend to try with different setups and see what's the best option. I might ping Bohdan as suggested too :)

Thanks for your replies,

Marc Serra
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
mserra
Posts: 4
Joined: Fri Nov 28, 2014 11:38 am

Mon Dec 01, 2014 3:57 pm

anton (staff) wrote:So it's "Compute and Storage Separated" and you plan to use iSCSI as a connectivity to VMware vSphere. OK :) We'll re-check current status of iSER with vSphere as a client.
Yes, that is correct. Sorry if I didn't leave it clear, I think I may have missplaced the thread and should have dropped it under StarPort instead of StarWind Virtual SAN :P
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Mon Dec 01, 2014 5:17 pm

No problem at all :) I don't have an answer right now but I'll have it quite soon :)
mserra wrote:
anton (staff) wrote:So it's "Compute and Storage Separated" and you plan to use iSCSI as a connectivity to VMware vSphere. OK :) We'll re-check current status of iSER with vSphere as a client.
Yes, that is correct. Sorry if I didn't leave it clear, I think I may have missplaced the thread and should have dropped it under StarPort instead of StarWind Virtual SAN :P
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
mserra
Posts: 4
Joined: Fri Nov 28, 2014 11:38 am

Tue Dec 02, 2014 10:22 am

anton (staff) wrote:No problem at all :) I don't have an answer right now but I'll have it quite soon :)
Great! I'm looking forward to it :)
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Wed Dec 03, 2014 3:40 pm

Please check your Inbox. I've took a few implementation questions off-line. Thanks!
mserra wrote:
anton (staff) wrote:No problem at all :) I don't have an answer right now but I'll have it quite soon :)
Great! I'm looking forward to it :)
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
Post Reply