Considering switching from FreeNAS

[Cabuzzi]

Hey everyone. First post here.

I've been a FreeNAS user for many years now, but am at the point where I am ready to try something new. The major catalyst for this has been going to MPIO iSCSI (quad port gbnic) and not seeing the utilization I expected... even with multiple operations. No matter what I do, XenServer only seems to want to read/write from one channel... even when migrating multiple VMs concurrently. I'm a Citrix employee and have seen XenServer blaze over iSCSI, but not with my implementations (lol). It obviously could be something I'm doing wrong, but I don't think so.

When I try and ask about it on the FreeNAS forums, Cyberjock there (very smart dude) is usually the first to reply with: "You need more RAM!" <paraphrased> or "CoW doesn't play well with iSCSI". Over the years, the amount of RAM I've had available has grown from 4GB (in an HP N40L microserver) all the way up to 64GB (Dell R710). The results have been the same, regardless... despite Cyber's insistence that RAM is the culprit. Maybe for a much larger environment... but we're talking about two Xen hosts and 1-2 FreeNAS boxes. Maybe there is a guide out there for FreeNAS to "tune" ZFS/iSCSI to enable MPIO to work the way it's designed, but I haven't found one. In fact, most FreeNAS forum requests for such assistance end up with the same thing. The end result of my troubleshooting/research is that:

1) It's possible, but not worth the hassle. Cyber has often indicated that you need to throw more hardware at the problem, as opposed to tweaking to get things working better.
2) MPIO w/iSCSI is inherently limited by FreeNAS and/or ZFS, and there really is nothing you CAN do.

So at this point, I am considering other alternatives. I will say, I LOVE ZFS. In my opinion, it is the best file system I've worked with. ZFS snaps have saved my *** many times, and being able to replicate over SSH has been a suitable alternative to FreeNAS not being capable of HA. Still, I am willing to try something else that may better suit my needs, and will work well with my hardware.

That need is essentially being able to better utilize the 4-gig NICs I use in my servers (or the ones built into the R710... Broadcom, I believe). I have roughly 25-30 machines MAX, spread across the two XenServers, along with two FreeNAS SRs... one per server, but have considered going to two SRs per R710. Since my switching environment supports 802.1ad, I have even tried LACP/NFS, but the XenServer gurus have indicated that MPIO iSCSI would be better, performance-wise

My hardware environment is as such:

- Four Dell R710s:
-- (2) with 64GB RAM and (2) Xeon X5570s (used for the Xen hosts)
-- (2) with 32GB RAM and (1) Xeon E5504 (used for the FreeNAS hosts)
- TP-Link TL-SG2424 layer 3 GigE switch with 802.1ad (LACP) support
- Integrated quad-port NIC in R710 (also tested with separate quadport HP NC364T Intel-based adapters... same results)
- I also have a Dell 2950 with 32GB of RAM (and two X5470 Xeons) and an HP NC364T that I was using previously, which behaved roughly the same in FreeNAS... It just used more power, and was MUCH louder.

Focusing on the two R710s I'm using for storage:
- Each has the aforementioned single Xeon 4-core E5504 and 32GB of ECC RAM
- One has a Dell Perc 6/i, with the six drives presented in RAID0 (which was my only option)
- One has a Dell R200 (per FreeNAS forum recommendations) to directly pass-through the drives
- Each has the latest firmware (with the exception of the H200 card)
- One has three 2TB 5400RPM Western Digital Red drives in a ZFSv1 pool, the other has three 2TB 7200RPM Hitachis
- Each has a 120-240GB SSD for L2ARC cache
- Each has a single file-based (not device) Xen SR presented to the two-host XenServer pool

Other notables:
- The Dell H200 is firmware version 7. FreeNAS gripes about it not being version 20, and I've also read that others have crossflashed the adapter to an LSI firmware, to be be able to truly passthrough the drives to FreeNAS as JBOD... not RAID0, and take advantage of SMART
- Both XenServer and FreeNAS are pretty much out-of-the-box deployments, with little customization
- Both XenServer hosts have their iSCSI software initiators setup with all four IPs
- Both XenServer hosts have multipathing enabled
- I know of no specific need (or way) to enable MPIO on FreeNAS. If this is something I'm missing, that might explain everything

So now that I've explained what I've been trying to achieve with FreeNAS, is Starwinds capable of this? Obviously, FreeNAS being "free" is a huge motivator for me, as this is just a home domain/lab for deploying Citrix products like XenApp, XenDesktop, NetScaler, etc. I also host my family's domain on a very similar environment, but my storage needs there are much lighter (though I would love MPIO working there as well.

[Cabuzzi]

Something I forgot to mention... the TP-Link switch is dedicated to storage only. This is also out-of-the-box with no settings tweaked, nor VLANs created for the individual subnets. There might be some 'noise' between the subnets, but I didn't see the need to further isolate the traffic, as only storage traverses the switch. It is completely separated from the internal management network, so I need to plug in a laptop to access it.

[Van Rue]

According to Xenserver forum, NFS and iSCSI have almost equal speeds even multipathed. Personally I think there is a limitations on any Unix solution and multipathing. Samba does not yet support multi-channel (But its been funded and someone is actively working on it.). Your post is the very reason I am leaning away from a Xen and Unix solution, although they perform wonderfully on the file and OS level, there is some "magic" required to get the network transfer speeds high. I too have tried and failed to get the formula right. Since 10Gb switches are out for some of us, MS has really done a good job with SMB 3.0 multi-channel in a 1gb environment. FC is another option, but again its tricky under Linux and support of different OS is limited, you may have to run an older version to get HW compatibility you need.

I am currently testing network cards right now with SMB under Windows 2012/Starwind and found that some of my older Intel Quad port cards don't really support RSS on SMB 3.0 (even though they say they do), so I have to upgrade some of my LAN cards before I follow this path. That is the first wall I hit. But even if SMB 3.0 does work well, Hyper-V will still move an OS over a single channel because of TCIP limits. On the cards that do support SMB 3.0 I have to tweak the advanced settings a bit away from the default, so its not as out of the box.

Personally, Windows 2012 R2 File Servers/Hyper-V is where I am leaning as well, but testing will continue after a NICs to arrive in a few days. Id set up your own test lab on your equipment to see, MS gives a generous 6 month trial which I have always appreciated. Pls share back here though, I am in the same boat as you.

[Bogdan (Staff)]

Hi Cabuzzi,

Speaking about Load balancing, I would like to mention that StarWind VSAN fully supports iSCSI multipath. This is a main statement we depart from. What it means is we care about performance as well as redundancy. Without any limitations, you could setup multiple paths between the client and storage server. The next thing has to be highlighted is the hardware agnostic approach which StarWind go with. Having such equipment described, you could easily implement at least free version of our product and enjoy SMB Multichannel technology completely. Also please note that StarWind doesn't consume much compute resources (CPU and RAM) (* you do not require any extra RAM or CPU to create and run regular thick provisioned imagefiles) for maintaining its functionality. Here you can learn more about our system requirements: https://www.starwindsoftware.com/system-requirements

[Cabuzzi]

@Van Rue
Thanks for your reply, and sorry to hear you are in the same boat. What you mention about iSCSI and MPIO (not NFS and LACP) is something I've heard often, though mostly in regards to Linux. FreeNAS runs on BSD, which is supposed to be very iSCSI and MPIO-friendly. That being said... XenServer is not BSD-based, which has led me to believe that it may possibly be the bottleneck. As a long time user of XenServer, and more recently as a Citrix employee, this pains me to admit. My testing (fiddling, actually) did not end after making this post. I have had some limited success in configuring FreeNAS in a way the XenServer seems to like, and that is by blowing away the file-based iSCSI extent I had created, and creating a block-based extent instead. Once I connected the SR to Xen, all four paths were used, although very sporadically. I then found a page with some multipath.conf settings that also helped, but instead of 100MB/s traveling over 1 GigE connection, I now had 25MB/s traveling over four.... despite the four connections being present on both ends (same brand NICs, even).

I have also considering the move to 10GigE, but have always been halted by the price of switching equipment. I've seen some clever solutions people have cooked up, and am somewhat interested again... as it's been a year since I last researched it. I have even seen people use regular 1GigE switches with four 10GigE SFP ports, and simply used those instead. Since I have four hosts (two Xen, two FreeNAS), that may actually work. I'm curious to know what you've come up with as an affordable 10GigE solution... or at least a reasonable priced one.

@Bogdan
I am still definitely interested in the StarWind solution. Once I get some old storage moved, I will have two Dell 2950s with 32GB of RAM each for testing. I may setup StarWind on one box, and try a variety of hypervisors on the other box, through their own switch, just to see how it performs. I'll be honest, it's going to be hard moving from FreeNAS, just because I've used it for so long... but I'll try (most) anything once.

[Cabuzzi]

Ok, so I finally blew away FreeNAS (really... I wanted to blow it up). But after reading guide after guide, I have not found one that tells you how to setup your disks initially. They all seem to jump into creating the server, creating the iSCSI target, then adding devices. In following those guides, I could absolutely do all that... but what I got was three independent drives, each assigned their own LUN. So when I tried to add it to XenServer, the three drives would present three different LUNs as a choice... with no option to set the three up as an array of any sort.

I searched around some more, but came to the conclusion (maybe erroneously) that StarWind, in and of itself, does nothing to actually build arrays of any sort. It maybe only presents a currently existing physical device to a target, and goes from there.

Following that assumption, I used Storage Spaces in Server 2012 R2 to build a roughly ~4TB available parity array from the three 2TB drive pool. However, when I go to add that new drive (I gave it a drive letter and formatted it to NTFS), then add it to the target, I get a yellow exclamation over the target, and a red exclamation over "DiskBridge1" (not understanding the StarWind terminology just yet).

So... here's a screenshot. Please tell me what I'm doing wrong.

[darklight]

You have to choose Hard Disk Device -> Virtual Disk instead of Physical Disk option while creating a Starwind device. And you will be fine.

My concern here is that AFAIK Storage Spaces is aligned to 4K block size and for XenServer you have to use 512B block size devices... not sure it will work this way but give it a try!

[Vladislav (Staff)]

Hi Cabuzzi!

These aren't the droids you're looking for (c)

In order to create your StarWind devices you should choose Virual Disk option in the "Add new device" manager. Please check the screenshot in the attachment

P.S: Our Xen documentation looks a bit outdated so I would like to advice you starting with our Hyper-V guide: https://www.starwindsoftware.com/techni ... manual.pdf in order to learn how to create StarWind devices and than move to Xen guide: https://www.starwindsoftware.com/techni ... Server.pdf after initial steps are finished.

P.P.S: Also please keep in mind that we are not yet officially support Storage Spaces and hardware RAID is highly recommended.

Also, thanks darklight for your awesome contribution. We highly appreciate it

[Cabuzzi]

You have to choose Hard Disk Device -> Virtual Disk instead of Physical Disk option while creating a Starwind device. And you will be fine.

My concern here is that AFAIK Storage Spaces is aligned to 4K block size and for XenServer you have to use 512B block size devices... not sure it will work this way but give it a try!

Thanks. I figured I was missing something obvious. As for XenServer, I've heard this come up before, but it has never been any real issues as a result. There can be some issues with alignment, so I'm told... but nothing too noticeable on my end.

There is a good discussion of it here on the XenServer forums. Tobias (XenServer god) gives a good rundown on it:
http://discussions.citrix.com/topic/360 ... e-in-6265/

At any rate, when I built the virtual disk, it allowed me to choose 512 or 4096, so I chose 512 to be safe (someone chime in if this is wrong).

Hi Cabuzzi!

These aren't the droids you're looking for (c)

In order to create your StarWind devices you should choose Virual Disk option in the "Add new device" manager. Please check the screenshot in the attachment

P.S: Our Xen documentation looks a bit outdated so I would like to advice you starting with our Hyper-V guide: https://www.starwindsoftware.com/techni ... manual.pdf in order to learn how to create StarWind devices and than move to Xen guide: https://www.starwindsoftware.com/techni ... Server.pdf after initial steps are finished.

P.P.S: Also please keep in mind that we are not yet officially support Storage Spaces and hardware RAID is highly recommended.

Also, thanks darklight for your awesome contribution. We highly appreciate it

So... I've made some progress. I had to uninstall/reinstall SW because no matter what I did, I couldn't get the the management console to connect after I removed the old iSCSI stuff I erroneously built. Also, it did not ask me for a license key this time, but I do have it on the desktop. The option of Host > Install License is grayed out, so I'm hoping all is good there. No nag screens yet.

As for Storage Spaces, I swapped my old JBOD controller out for a Dell Perc 6/i adapter. I hooked it up, removed all the old cables, installed the battery, and setup a RAID5 array with three 2TB drives with WB enabled.

I only have a few lingering questions at this point, before I attach it to XenServer and try moving a test machine over to it.

1) Is MPIO enabled by default? I've reserve all four ports of the quad port NIC on the R710 for iSCSI traffic.

2) What (if any) changes should I make to the multipathing on the XenServer side? For FreeNAS, I had to put an entry in the "devices" section of the multipath.conf file. It would work without it... it's just MPIO wasn't working *great* until I added the FreeBSD section. Here is a look at what the config looks like:

Code: Select all

[root@xen1 etc]# more multipath.conf
# This configuration file is used to overwrite the built-in configuration of
# multipathd.
# For information on the syntax refer to `man multipath.conf` and the examples
# in `/usr/share/doc/device-mapper-multipath-*/`.
# To check the currently running multipath configuration see the output of
# `multipathd -k"show conf"`.
defaults {
        user_friendly_names     no
        replace_wwid_whitespace yes
        dev_loss_tmo            30
}
devices {
        device {
                vendor                  "DataCore"
                product                 "SAN*"
                path_checker            "tur"
                path_grouping_policy    failover
                failback                30
        }
        device {
                vendor                  "DELL"
                product                 "MD36xx(i|f)"
                features                "2 pg_init_retries 50"
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                path_grouping_policy    group_by_prio
                failback                immediate
                rr_min_io               100
                path_checker            rdac
                prio                    rdac
                no_path_retry           30
        }
        device {
                vendor                  "DGC"
                product                 ".*"
                detect_prio             yes
                retain_attached_hw_handler yes
        }
        device {
                vendor                  "EQLOGIC"
                product                 "100E-00"
                path_grouping_policy    multibus
                path_checker            tur
                failback                immediate
                path_selector           "round-robin 0"
                rr_min_io               3
                rr_weight               priorities
        }
        device {
                vendor                  "IBM"
                product                 "1723*"
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                path_grouping_policy    group_by_prio
                failback                immediate
                path_checker            rdac
                prio                    rdac
        }
        device {
                vendor                  "NETAPP"
                product                 "LUN.*"
                dev_loss_tmo            30
        }
        device {
                vendor                  "FreeBSD"
                product                 "iSCSI Disk"
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                rr_min_io               1
        }
}

3) During the install, I was asked if I wanted to do thick provisioning, or LSFS. I didn't previously know what that was, so I looked it up on your site, and it seemed to be storage optimized for virtualization, so I choose that.

So that's where I'm at. Here's a picture of my current setup:

[Cabuzzi]

So, I found this page after some google search tweaks (to filter only from this year) and found this article:
https://knowledgebase.starwindsoftware. ... correctly/

It seems to end abruptly, however, after installing enabling MPIO in Windows for iSCSI devices. Not sure what to do next...

[Cabuzzi]

I've read through both the VMware and Xen docs a few times now, but I'm still confused. This is probably because when setting up FreeNAS on BSD, I only "bound" the four IPs I used for iSCSI on the server under the IP settings for the portal. If I understand StarWind correctly, I actually have to setup the Microsoft iSCSI initiator (which you would normally connect to a remote iSCSI target??) to point back to the target running on the same server? This is what I'm stuck on now... and I can't seem to find the right documentation to understand what I'm doing wrong. What further confuses me is the references to HA in the VMware document, which I won't be able to use (since I'm running the free version to start with).

[darklight]

To get a general idea of what you are actually trying to achieve, please refer to following guides:
https://www.starwindsoftware.com/starwi ... ng-started - to get the basic understanding of how it actually works.
https://www.starwindsoftware.com/techni ... FS_NFS.pdf - to proceed with HA storage for XenServer.

[Cabuzzi]

I appreciate the reply, but you just replied with documentation I had already read. I came across the stuff I linked in the post before because I was trying to make multipathing work.

I can connect to the iSCSI share. Sorry if I did not establish that before. But I cannot get MPIO working. I get traffic on one interface only. In my experience, this is because round robin is not set in XenServer, which requires an entry in the multipath.conf file, a section that Citrix says is "vendor-provided". This is not something I can guess on, and it prevents me from allowing Xen to round robin on that SR (see question #2 below).

2) What (if any) changes should I make to the multipathing on the XenServer side? For FreeNAS, I had to put an entry in the "devices" section of the multipath.conf file. It would work without it... it's just MPIO wasn't working *great* until I added the FreeBSD section. Here is a look at what the config looks like:

Code: Select all

[root@xen1 etc]# more multipath.conf
# This configuration file is used to overwrite the built-in configuration of
# multipathd.
# For information on the syntax refer to `man multipath.conf` and the examples
# in `/usr/share/doc/device-mapper-multipath-*/`.
# To check the currently running multipath configuration see the output of
# `multipathd -k"show conf"`.
defaults {
        user_friendly_names     no
        replace_wwid_whitespace yes
        dev_loss_tmo            30
}
devices {
        device {
                vendor                  "DataCore"
                product                 "SAN*"
                path_checker            "tur"
                path_grouping_policy    failover
                failback                30
        }
        device {
                vendor                  "DELL"
                product                 "MD36xx(i|f)"
                features                "2 pg_init_retries 50"
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                path_grouping_policy    group_by_prio
                failback                immediate
                rr_min_io               100
                path_checker            rdac
                prio                    rdac
                no_path_retry           30
        }
        device {
                vendor                  "DGC"
                product                 ".*"
                detect_prio             yes
                retain_attached_hw_handler yes
        }
        device {
                vendor                  "EQLOGIC"
                product                 "100E-00"
                path_grouping_policy    multibus
                path_checker            tur
                failback                immediate
                path_selector           "round-robin 0"
                rr_min_io               3
                rr_weight               priorities
        }
        device {
                vendor                  "IBM"
                product                 "1723*"
                hardware_handler        "1 rdac"
                path_selector           "round-robin 0"
                path_grouping_policy    group_by_prio
                failback                immediate
                path_checker            rdac
                prio                    rdac
        }
        device {
                vendor                  "NETAPP"
                product                 "LUN.*"
                dev_loss_tmo            30
        }
        device {
                vendor                  "FreeBSD"
                product                 "iSCSI Disk"
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                rr_min_io               1
        }
}

Also, regarding question 3, I just wanted confirmation that LSFS is the correct file system to use for VMs on shared storage (vs NTFS).

3) During the install, I was asked if I wanted to do thick provisioning, or LSFS. I didn't previously know what that was, so I looked it up on your site, and it seemed to be storage optimized for virtualization, so I choose that.

[Cabuzzi]

I'll go ahead and reply again, even though I get the feeling you guys are ignoring me because you think I'm not looking up documentation.

If that is the case, there are two problems with that:
1) Your Xen documentation is old. If you want to say you support it, then it really needs to be updated.
2) Because of how old the Xen documentation is, the multipath.conf entries do not match how Xen detects your storage over iSCSI.

Referring to the following document at: https://www.starwindsoftware.com/techni ... Server.pdf

To apply multipathing take the following actions for each XenServer in the pool:
1. Insert the following block into the defaults
section of the /etc/multipath.conf:

Code: Select all

defaults {
	polling_interval 10
	max_fds 8192
}

2. Insert the following block into the devices section of the /etc/multipath.conf:

Code: Select all

device {
	vendor "ROCKET"
	product "IMAGEFILE"
	path_selector "round-robin 0"
	path_grouping_policy multibus
	getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
	prio_callout none
	path_checker readsector0
	rr_min_io 100
	rr_weight priorities
	failback immediate
	no_path_retry 5
}

...but when you look at the device paths in XenServer, you'll see that "ROCKET" and "IMAGEFILE" are not even how the paths are presented to Xen. It shows in Xen as "STARWIND,STARWIND". If I were to enter what is in your current documentation, the multipath policies would not apply. If I'm not mistaken, this is a change on your end... not Citrix's, but I could be wrong. Either way, I have to wonder if the other settings defined for the device are still correct.

Code: Select all

[root@xen1 by-scsid]# multipath -ll
2e0cdb9916453fd4f dm-12 STARWIND,STARWIND
size=3.5T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 109:0:0:0 sdf 8:80  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 110:0:0:0 sdg 8:96  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 111:0:0:0 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 112:0:0:0 sdi 8:128 active ready running
36589cfc000000b8c7d2eba0fc493b9e8 dm-0 FreeBSD,iSCSI Disk
size=7.0T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 48:0:0:0  sdb 8:16  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 49:0:0:0  sdc 8:32  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 50:0:0:0  sdd 8:48  active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 51:0:0:0  sde 8:64  active ready running

I'm willing to give Starwinds a chance, and even license it as this test domain grows (already up to 10 rack servers... and looking to move to blades). I also work in consulting for Citrix and DoD, but I cannot recommend a solution at this point. Your documentation is just too "all over the place".

Thanks, and hopefully my post doesn't rub you the wrong way.

[Vladislav (Staff)]

Hi Cabuzzi!

Ouch! It looks like I have provided a really old guidance for you and I am really sorry for that

We actually have a new one available here: https://www.starwindsoftware.com/techni ... r-Pool.pdf

Please refer to "Applying multipathing" section of this doc and let me know if it helps.

Meanwhile I am configuring a test setup in my lab in order to double check it.

Thank you and sorry for this silly mistake