Caching with CVM

[altmannj]

Hello,

Is it possible to configure a caching solution, whether native to the CVM VSAN appliance or via back-end with direct PCI access? I have bcache configured in Proxmox and have that volume presented to CVM as a disk, however, this is virtualized and adds a small I/O hit due to non native drive access. I wasn't able to configure this when passing PCI directly to the CVM appliance as it appears the CVM UI only resolves kernel based raw disks and no mapped disk like /dev/bcache0.

[yaroslav (staff)]

Hi,

If you are referring to using PCI SSDs as cache, the answer is "no, please don't do that".
Write back RAM caching for SSDs does not make much sense, IMO, and it might make highly available devices a bit touchy, causing full sync or mutual not synchronized every time the restart procedure is mishandled. Write-through caching boosts reads, but in highly available mode, it does not make sense, as client reads from all replication partners, so reads should be OK.
Same for SSD caching, which is available only in write-through mode. Sure, you can try using StarWind HA SSD caching, but that cache boosts only reads, which IMO is a waste of SSDs' potential, as reads are mostly OK due to reading from all replication partners. It is much better to put StarWind highly available devices on top of them to have both reads and writes improved.

p.s. Use pass-through or raw device mapping in Proxmox for data volumes. QUEMU, even with write-back cache, might eat some performance.
p.p.s. If it is for lab use only, try installing opencas inside CVM and give this kind of hardware caching a try. This one should be good if configured right. Again, this is ONLY for lab use, as data loss might be possible in case of hardware failures if that volume is misconfigured.
And, last but not least, if you want to squeeze the maximum performance of your system, join our technical preview for NVMe-oF https://www.starwindsoftware.com/techni ... ew-program.

Good luck with your project.

[altmannj]

Thank you for the information! This is for a home lab. I have two higher performing SSD's that are write-intensive drives with storage SSD's that are read-intensive hence the desire for utilizing the write-intensive drives for cache. I had run the OpenCAS setup within Proxmox itself and presented that raw volume to the CVM guest VM and had this running in a two node + qdevice configuration with no issues for some time. Currently, OpenCAS does not support Proxmox's newer kernel, 6.17.

I figured that if CVM can't see bcache drives that it probably wouldn't see OpenCAS drives, but I can give it a try.

[yaroslav (staff)]

Thanks for your update and good luck with your project (you can ask someone technical about opencas if you are a part of beta, I think).
I doubt anyone tested bcache on our side tho.

[altmannj]

Is it possible to use OpenCAS with the controller virtual machine? If i create a caching device and a backing device the CVM recognizes the physical disks as "cas1-1 (cache)" and "cas1-1 (core)" for both the caching and backing devices. It's then listed as type "OpenCAS" with the name "cas1-1" under pools. It further lists an unmounted volume under volumes. I'm not sure if the CVM can recognize the volume even if you put a ext4 on /dev/cas1-1 and mount it.

I've further tied adding cas backing to both an MD raid as well as an LVM logical volume, all of which won't work.

From what I gather, CVM requires a strict block listing of device -> partition 1 -> LVM partition 1 > LVM volume in order for the UI to see all the way up to the volume level to be consumed by the LUN.

The only way I can conceivably get caching to work is to create the OpenCAS (or bcache) on proxmox and map the raw volume to the CVM vm.

Thanks!

[yaroslav (staff)]

Hi,

Yes, passing through individual disks for cache and controller for core is the preferred way [and the only tested one, I believe]. Then, I think, you can create an LVM for each so that the individual disk issue does not affect the pool.
I cannot advise on more detailed OpenCas configurations in CVM here, I am afraid. Still, OpenCas should work in the CVM.
bcache was not tested, however, for lab use, it seems a fun scenario to test.
Please note that OpenCAS is quite a sensitive thing. It might not handle discards well, in particular, causing storage operations to hang badly, which results in StarWind VSAN hangs. Again, that's not for production [you need a support tech to glance in the configuration if you want to move it to production], yet OpenCas is a fun thing to poke around with. I enjoyed it myself, I'd say.

[akena]

Hi,

If you are referring to using PCI SSDs as cache, the answer is "no, please don't do that".
Write back RAM caching for SSDs does not make much sense, IMO, and it might make highly available devices a bit touchy, causing full sync or mutual not synchronized every time the restart procedure is mishandled. Write-through caching boosts reads, but in highly available mode, it does not make sense, as client reads from all replication partners, so reads should be OK.
Same for SSD caching, which is available only in write-through mode. Sure, you can try using StarWind HA SSD caching, but that cache boosts only reads, which IMO is a waste of SSDs' potential, as reads are mostly OK due to reading from all replication partners. It is much better to put StarWind highly available devices on top of them to have both reads and writes improved.

p.s. Use pass-through or raw device mapping in Proxmox for data volumes. QUEMU, even with write-back cache, might eat some performance.
p.p.s. If it is for lab use only, try installing opencas inside CVM and give this kind of hardware caching a try. This one should be good if configured right. Again, this is ONLY for lab use, as data loss might be possible in case of hardware failures if that volume is misconfigured.
And, last but not least, if you want to squeeze the maximum performance of your system, join our technical preview for NVMe-oF https://www.starwindsoftware.com/techni ... ew-program

Good luck with your project.

Thank you very much for sharing. I need it.

[yaroslav (staff)]

You are welcome