Guidance on my my setup

[gorb]

I would like some assistance properly configuring 3 nodes. Two nodes being used as replicated vSANs and the 3rd node being used as a vSAN providing backup storage for DPM 2012.

About my environment:

We need to provide around 65TB usable on each active SAN. We're heavy read (75%+) and we will be using Starwind as storage for a Hyper-V CSV, iscsi targets for linux hosts, and for network user shares.

Hardware/Software:

OS: Windows Server 2012 R2
Starwind: Latest stable v8 build
Backup software: DPM 2012

Chassis: 3x 36 drive bay SMC w/ SAS Raid controllers with 512mb cache and BBU
Ram: 192GB per node

HDD: HGST 7k4000 SAS 4TB for storage nodes, Seagate Constellation 3TB Sas and Segate 3TB SATA in external storage attached in the backup node
SSD: 2x Intel DC P3500 2TB PCIE NVME SSD (could add 2 more if needed)

Network: Infiniband 40GB switch / connect x2 dual port cards in each host and 10gb Ethernet

Here is my Plan:

Each primary storage node will be setup with Raid 10 w/ cache enabled on the Raid card. The backup node will be Raid 6.
All ram not used by the base OS will be dedicated to L1 Cache in Write-Back mode
One PCIE SSD in each node as dedicated L2 Cache in Write-Back mode
SMB 3.0 for the MS CSV
iScsi for other storage presentation

Some things I would like clarification on:
I've heard that L1 Cache should be at a ratio of at least 1GB per TB on each node. Is this correct? Does adding more than 1GB per TB improve performance even more?
I've heard that L2 Cache should be 10-15% of your total available drive space. Since I only have 2TB of SSD per node is it worth even using this feature?
What happens if a SSD (L2 Cache) fails while the nodes are in production? Is it unsafe the run a single SSD for cache in each node?
How big of a performance increase is write through vs write back?
Finally, anything else you think needs to be discussed to optimize this setup please pass it this way. Thanks so much!

[gorb]

Do I need to contact support for this type of assistance?

[darklight]

Hi, Gorb.

Pretty nice and reasonable plan. Only a few thoughts from me:

I believe it is not possible to enable L2 cache in write-back mode anymore. Only write-through improving reads. Furthermore, I noticed some strange performance problems with L2 cache on traditional SSD drives. Read performance was slower than a local storage without any caching. On the other hand, PCIe flash drives do the job boosting the whole storage read performance almost to the speed of flash, which is good. So I decided to use my SSDs as a fast HA device for performance-critical VMs.

The other point is SMB share. Usually, I place Hyper-V VMs directly to CSV avoiding installation of an additional instance like SoFS. It is a must for StarWind Free version since it does not support direct iSCSI connections, but for a full version I prefer to set the things up as simple as possible. That's why i enable SoFS only for network file shares/whatever.

AFAIK 1GB for 1TB is a minimum L1 cache size recommendation from StarWind. I tried to to assign RAM according to this minimal recommendation and even more, like 10GB for 1TB, but it does not influence my performance too much. The only way to get L1 cache performing as hell is to cover your "working set" with RAM. For virtualized environments, that should be like 10% to 30% of total working (used by VMs) storage you have. It took some time to experiment and play with the values. If you can not afford such a large cache then the performance difference between 1GB and 5GB per 1TB is actually not so big.

Anyway, the best method to get the best performance is testing/benchmarking since sometimes your hardware can show very weird things.

[gorb]

Thanks Darklight.

I was hoping to get some Starwind feedback on that config too. You're experience is helpful, but I would really like some best practices guidance from the developers.

Specific to your comments. Did you say that you saw no positive impact from adding L2 Cache? Is that typical? Also, in regards to the ram. Since the array is going to be 72TB, it's going to be impossible to do 10gb to 1tb.

[darklight]

L2 cache behavior is very dependent on underlying hardware. It does improve performance on *really* slow disks and may improve it on faster ones. Depends on spindles. And on SSD vendor and speed. The only advice I can give you is testing.

With 1GB per 1TB you are already good enough.

[Tarass (Staff)]

Hi guys, thank you very much for your contribution.

I have to admit that L2 cache still has some minor performance issues on some specific hardware configurations that will be improved in an upcoming build expected by the end of the year.

[DilanRoster]

thanks for this guidance on the setup! everything is understandable!

[Michael (staff)]

Thank you as well!