OK, now goes StarWind L1/L2 Vs. MSFT CSV/SS WB cache (keep in mind I'll talk clustered configs only, Storage Spaces features for clustered and non-clustered setups do differ!)
L1:
MSFT has limited-purpose block cache for CSV (anything that does not belong to CSV cannot be cached by it obviously). Initially MSFT had created CSV cache targeting CSV consuming apps (SQL Server and Hyper-V mostly) as they deal with VHD(X)s and DBs accordingly in a so-called "pass-thru mode" (file system cache provided by Windows Cache & Memory Mgr is bypassed and all I/O goes immediately to/from disk) for data integrity purposes. To mitigate a complete lack of a file system cache MSFT had brought CSV cache to table. Compared to file system cache is has one but major drawback - it's READ-ONLY. So it's does accelerate READs but has nothing to do with WRITEs. Moreover: it not only does not accelerate writes but it also does nothing to adsorb writes to keep underlying flash happy - all I/Os go immediately to flash basically burning it out (this is absolutely **CRITICAL** and is true for VMware Virtual SAN as well, that guys also don't use RAM as L1 cache and target all initial I/Os to flash only). Also CSV cache is not synchronized between hypervisor nodes so VM migrated from HostA -> HostB would start from a "cold" state which obviously affects performance. You'll notice this with every I/O hungry VM. CSV cache is not deduplicated even if served content has dedupe enabled on it. Results waste of RAM and poor ROI as same data blocks are stored in memory multiple times (for a file system cache MSFT has an ability to pinpoint dedupe buckets to memory so it's CSV cache problem and not MSFT problem in general). Hash protection is not used. You can have bitrot with CSV cache and you'll never know this. Even usin ReFS with integrity streams don't help here (there's no way to use ReFS data integrity with running VM images either way...)
StarWind has distributed write-back L1 cache. We don't care what you layer on top of it - Hyper-V CSV, cluster-aware file system like VMFS or expose raw LU to Oracle DB. We do support everything. Our cache does accelerate both reads and writes. Our cache is designed to adsorb writes and keep underlying flash (L2 or primary storage) as happy as possible with reduced amount of writes (same block overwritten multiple times would go once to flash with StarWind but as many times in MSFT case as app was actually writing it). We're synchronized between multiple nodes, keep content coherent so we're a) safe (there are multiple replicas of the same cached data) and b) VM moved from one hypervisor to other always starts from a "hot" state with all the cached data already in memory. Our L1 cache is deduped (if paired with LSFS as underlying storage of course and it also has dedupe enabled) so we can store in cache memory more data then normally expected. StarWind uses strong 256-bit signatures for cache blocks so if cache content is damaged we'll dynamically take cache away and replace with a data from an underlying nodes.
L2:
MSFT requires you to use SAS-attached flash to WB cache. SAS is a very poor choice here as it's both expensive (compared to commodity SATA SSDs) and slow (compared to Enterprise PCIe cards). Want to use huge loads of cheap and well-performing SATA SSDs from Intel for cache? MSFT cannot do it. Want to use that 1M IOPS 1TB Fusion-IO card for cache? MSFT cannot do it either. Next... SS Write-back cache can be configured once when building storage space. You cannot throw in more flash later to WB existing LU, you cannot remove cache from LU you don't want to cache anymore either. Cache size cannot be changed as well. You provide a settings for the pool but then virtual disk is created you're done. You cannot have cache settings being different from pool settings. So if you have mirrored pool - you have to go mirrored cache in a pool. So very inflexible... With Clustered Storage Spaces all nodes have access to the same content so caches are "distributed" (well, just because you have many paths to the same content) but again like with CSV cache MSFT has no dedupe option for WB cache. No hash protection.
StarWind can use any type of memory for cache. So you may have all-SATA primary storage pool for capacity paired with a PCIe-attached flash card for performance (that's actually recommended config for us). We can add / remove flash to LU any time you want. See need more IOPS? Throw in more cache. LU now handles ice cold data? Take away L2 cache and use it to accelerate something else. We can pair any types of cache protection with any underlying RAIDs. So you can use simple (no mirror) cache with RAID5/6 on a primary storage. StarWind L2 cache is distributed between multiple nodes so is safe to use and fast (number of I/O paths increase with every new replica added). Full dedupe ability when paired with dedupe-enabled LSFS config. Protected with hash sums.
As a side note we do provide users an ability to select between multiple policies "Write-Back for L1 and Write-Back for L2" Vs. "Write-Back for L1 and Write-Thru for L2". Second options uses RAM to accelerate reads and writes and adsorb writes and uses flash to accelerate reads keeping flash as healthy as possible.
So as you can see we beat MSFT on caching all-around. With V8 we don't recommend to use either CSV cache or SS WB cache. Use StarWind built-in stuff.
Klas wrote:Great initiative
What L1 cache size is recommended
What L2 cahce size is recommended
How many sync channels is optimal for X bandwidth. Example. 1 x 10 GBit iSCSI need 4 x 1 GBit sync. (In my lab with v6 and v8 beta I found that >3 x 1 GBit sync channels gave better performance than 1 x 10 GBit sync).
L1 and L2 cache contra Microsoft Storage spaces write back cache.
