Usage of CX4 interfaces for better 'LAN' performance ?

[JLaay]

Hi all,

Read about iSCSI storage/performance in a couple of (sub)forums as a more affordable solution for SMBs.
I stumbled on CX4 and although more expensive than iSCSI 1Gb it looks to me this could be a good solution to overcome the LAN performance issue. This in combination with a CX4 capable switch like the HP Procurve 2910al/5406zl switches.....
Leaves a good storage box with fast drives.

Any pros/cons ? Or just a stupid Idea

Thanx

Jaap

[Aitor_Ibarra]

CX4 - or any 10Gbit/sec Ethernet connection - is a great boost to performance, providing you've got fast enough drives to benefit. If any of your iSCSI targets need to break the 1Gbit limit, then you will need fast drives, or a striped RAID set, but 10Gbit is also useful if you have lots of busy targets on one Starwind box., even if the initiators only have 1Gbit.

Also, when I've looked at pricing, 10Gbit compared to 1Gbit with managed switches is nearly the same price or sometimes cheaper on per Mbit basis, i.e. 10x 1Gbit ports on a switch, server, and cabling will work out more expensive than a single 10Gbit. Also it's a lot neater in terms of cabling!

CX4 is great but seems to be being superceded by SFP+ which is a slightly smaller connector, so makes higher density switches and NICs possible. E.g. I have dual port CX4 Nics in my starwind boxes, but there's at least one quad port SFP+ NIC available... Also I have heard big hints that Supermicro are going to be doing SFP+ on the motherboard when Tylersburg chipsets come out (my hyper-v nodes are Supermicro twins with CX4 on the motherboard).

HP have a nice 6 port CX4 switch which is upgradeable to 8 ports, but I found it cheaper to get two 24 port Dell 6224s with a couple of CX4 modules in each (so 24 x 1Gbit and 4x 10Gbit per switch). Now Dell have just launched a 24 port SFP+ based switch, and it's cheaper per port...

Not sure if 10Gbit over cat 6e/7 (RJ45 copper) is going to take off; it requires higher voltages to avoid noise problems so power consumption is higher than CX4/SFP, but it can do longer runs (with SFP+ you can pay $$$$ and go optical for long runs),

[JLaay]

Hi Aitor,

Thanx for your reply.
I 'm really surprised there are not more reactions to my post.

I 'm looking for a solution for these SMB situations:

Within 1 building (all copper):
Room 1 > VMware host + SAN; room 2 > 1 VMware host + SAN.
SAN = Starwind HA. Devide data-hungry applicatiion over the two SAN with the respective VMs on the VMware host in the same room.

SAN: 1 dual CX4 interface and 1 dual 1Gb interface
(CX4 > VMware host; to switches: 1x 1GB port to other SAN for synchronization, 1x 1Gb port spare or redundancy SAN synchronization)
VM host: 1 dual CX4 interface and 1 quad 1Gb interface
(CX4 > VMware host, 1x 1GB port to both switches, 1x 1Gb VMware HA, 1x 1Gb spare (maybe VMware FT)
Switch 2x (vendor,type), x 1Gb ports for workstations,
same room: 2 x 1Gb uplink (team) to other switch in for user LAN
to other room: 2x 1Gb uplink (team) for user LAN, 1x 1Gb uplink for SAN synchronization, 1x 1Gb uplink for VMware HA, (1x 1Gb uplink VMware FT)

With 2 buildings (Fiber):
Idem hardware except for connection between locations.
In this situation my question is: are 2x 10Gb fiber redundant connection, with vlan configuration and maybe QOS, a secure solution with regards to SAN synchronization, VMware HA and FT?

Working a lot with HP switches, e.g. 2910al and 5406zl, prices for 10Gb Gbic tranceiver are ouch € 2000 ex! Will need four of them in this situation.
Looking at 'your' Dell 6224 with one SFP+ module + two 10gb SFP+ transceivers I come up to € 2600-3000 ex depending on fiber SM/MM and short/long range!

Thanx Jaap

[nbarsotti]

Just thought I'd throw my $0.02 in.
I think the 10GSPF+CU is going to over take 10GBASE-CX4 due to physical size restrictions, and you are correct about 10GBASE-T over Cat6/7 not getting much traction in the marketplace due to power requirements. SPF+ and CX4 also have a latency advantage over 10GBASE-T so they seem better candidates for iSCSI and FCoE.

With that said I am awaiting delievery of a new iSCSI server with the Intel 10g Dual SPF+ card and two other matching cards for my ESX servers. I have small shop and have the ability to directly connect my servers to the starwind server, there by not needing a 10GB switch and saving lots of money.

[JLaay]

Hi nbarsotti,

What brand/type of iSCSI server are you going for?
How many concurrent users do you have?
What kind of applications are they using?
Kind of usage?

If I'm correct you have one storage server.
Besides the money what arguments do you have not going for HA?


Thanx Jaap

[nbarsotti]

We are getting an R515 from Silicon Mechanics. Silicon Mechanics essentially resells SuperMicro chasis and motherboards. We only have about 30 concurrent users, all VMware ESX virutal machines. About 20 machines with a mix of SQL, IIS, Exchange, File and Print. The SQL servers get hit pretty hard.

The only reason I didn't go HA was cost. As I understand it HA will only be as fast as the slowest of your two servers, and I couldn't not afford two servers with 10gbe and SSD.

[JLaay]

The mix of machines in your shop seems to me a 'normal' mix within Windows environments.

Besides redundancy with HA, I was thinking using HA also as an option ( during normal operation ) to use it to devide the workload as evenly as possible.
You have to have the same storage space for redundancy on each storage, but each storage could be less 'rich' equipped.
Then if one storage fails, you stil have redundancy. With performance degradation of course. Depending how long it will take to get the failed storage running and synchronized again, this might be accepted by management because you can still work. Management then could decide which tasks should prevail and kick some users off .

Topics to be checked:
- MPIO method > fixed?
- MPIO failback > automatic/manual ? (Failback only after full synchronization )
- ....
- procedures in place

I 've put this question to support on how they think about it. No answer yet

Greetz Jaap

[nbarsotti]

I think HA will only marginly divide your work load. Every write to one server will happen concurently on the other server. depending on the number of NICs you use it could decread utiliation on the network.

[JLaay]

Yes you are right of course about the write io synchronization.
Still what I 'm curious about is the usage of applications in your shop and any other for that matter.
E.g what causes the SQL hits. Are these a result of queuries, report generation etc?
These will also do some write actions to be able display/print the result to the respective users, causing synchronization, workload on the other storage server
But in my 'mindset' this will be significant less than pure rapid concurrent data-entry by lots of users.

Greetz Jaap

PS I 'm not trying to be stubbern, just want to understand and learn

[Constantin (staff)]

First of all, sorry for such long time response!
Automatic failback functionality will be implemented in StarWind 5.5, which probably will be released at the end of January.
Also I would to ask you about your result configuration.

[JLaay]

Hi Constantin,

I don't know exactly what you mean with 'Also I would to ask you about your result configuration.'

I 'm evaluating SW and do a lot of reading about iSCSI storage. And backup / restore.
I am trying to find out if my 'thinking'of devided primary tagets is realistic.

Thanx and greetz

Jaap

[anton (staff)]

In the most of the configurations even with 1GbE it's disk subsystem being on the slow side. Not network.

[JLaay]

Hi Anton,

True it is especially the disksubsytem to worry about when it comes to throughput.
I 'v got a light Storage test setup, onboard raid 0 (dangerous I know ) with 4 WD RE3 500gb, systemdisk for W2K8 with SW excluded, with Gb switch to ESXi4 host.
With IOmeter with Max Throughput-100%Read > 96 Mb; RealLife-60%Rand-65%Read > 10,4 Mb.
With a thereoratical max throughput of 128Mb on a 1Gb link, I think I can say network is properly configured but the storage is not up for 'it'.

The 'it' however is the relative part. And depends on the shop, usage of applications, exceptance of 'the 'performance' etc.

Greetz Jaap

[anton (staff)]

First of all 1GbE is not 128 MB/sec . It's close to 110MB IN SINGLE DIRECTION in real life. So when mixing both reads and writes over the same cable you can get something close to 200MB/sec pretty easy (150-160MB/sec for low end NICs). But! Storage stack on the initiator, network stack on both target and initiator and storage stack on the target they do build a pipeline all together. So good bandwidth is something we could have only under heavy load as with pulsating traffic we'll have to load complete pipeline from the very beginning again and again and this results increased latency and bad numbers. That's why disk subsystem should be not matching network link but rather performing much better to compensate pulsations.

[JLaay]

Hi Anton,

Brr.. Bear with me because I am trying to understand this

So it's :

all the building up and tearing down links (on different levels) responding to seperate requests
and/or
the abilty of the disksubsystem to respond to those seperate requests, causing read/writes

that's causing latency and therefore bad numbers?

Greetz Jaap