Proxmox 2 node NVME-of Testing feedback

Test and discuss technical preview builds
Post Reply
nick.smith
Posts: 4
Joined: Thu Jan 29, 2026 10:40 am

Mon Feb 16, 2026 3:52 pm

Testing Environment

Hardware
2x Node with the following specifications

PowerEdge R6515
AMD EPYC 7302P 16-Core Processor
128GB of RAM
Logical Processors: 32
NICs: 8
Dual embedded Gigabit Ethernet
Broadcom 57414 Dual Port 25GbE SFP28 LOM Mezzanine Card
Mellanox ConnectX-5 EN 25GbE Dual-port SFP28
Mellanox ConnectX-5 EN 25GbE Dual-port SFP28


8x 1TB SATA SSD - connected to a Raid controller PERC H740P Mini configured in RAID-6 to give 5.3 TB of usable space
2x Dell Express Flash PM1725a 1.6TB SFF (Samsung NVME drives) - One for Proxmox OS and one for NVME fast VM storage
Dual power supplies


Operating System
2x Proxmox 9.1.5 nodes - Kernel Version: Linux 6.17.9-1-pve (2026-01-12T16:25Z)
1x Ubuntu-22.04.5 external Qdevice for Proxmox quorum vote

All setup in a Proxmox cluster 2x full Proxmox nodes with a Qdevice

Starwind NVME-of Setup
Followed the guide on this page
https://www.starwindsoftware.com/resour ... ng-web-ui/

Installed StarWind VSAN CVM KVM on each Proxmox node.
Version: Core StarWind Virtual SAN v8.0.0 (Build 20082)
Installed version: 1.7.788.7573

Starwind Networking setup
Image

I was only able to reach the full 25Gbit with PCI passthrough for the Replication nic but for Data nics as they need to be available on both Proxmox host and Starwind VMs.
We have tested SR-IOV to passthrough the data nics and still be able to use them on the Proxmox host, but the Starwind VM keeps disabling it every time after reboot and failed to do ping test during HA configuration.

This is the state of the Data nics after a reboot of the Starwind VM, they keep getting marked as Disable network interface.
Image

This meant testing we had to revert back to a virtual bridge and use virtIO NIC drivers for Data nics, which meant they were only showing as 1Gbit nics.
Image

While following the guide mentioned above it depicts this setup as a 2 node cluster and even shows you how to setup a Qdevice for Proxmox but when it gets to storage Witness is skips over this section, also the Starwind CVM does not currently have this functionality.

Extracted from guide -
“6. On the Failover strategy step, select the preferred failover strategy and click Next.
NOTE: The failover strategies for a LUN can be Heartbeat or Node Majority. In case of 2-nodes setup and None Majority failover strategy, Node witness (requires an additional third witness node), or File share witness (requires an external file share) should be configured. These options are determined by StarWind Virtual SAN license and setup configuration. Below are the steps for configuring the Heartbeat failover strategy in a two-node cluster.”
Is this something that is not available with technical preview license?


With the Node Majority option greyed out
“The feature will be provided in the next version”
Image

We proceeded to set up the storage via the “Heartbeat” strategy.

The setup was straight forward and we were able to import as shared storage into Proxmox after it was configured.
Image

During benchmark testing we got some impressive results via Windows VM hosted on the Starwind shared NVME-of storage
Image
32Gb file
SEQ1M Q8T1 - Read 19794.90 MB/s Write 3741.68 MB/s
SEQ128K Q32T1 - Read 9691.58 MB/s Write 4227.26 MB/s
RND4K Q32T16 - Read 912.20 MB/s Write 166.62 MB/s

These numbers seem way too high as a simple data copy in Windows would stall and speeds be stuck at zero for a few moments before picking back up again
Examples below
Image
Image

Testing some failover scenarios exposed some features that are currently not available in the technical preview.
For example being able to put LUNs into maintenance mode or mark a node a synced
Image

or mark a node a synced
Image

Only way to view the status is to Manage LUN to see what state it is in
Image
Image

NVME-of does look promising but some of the issues and missing features are putting some doubt in our minds to be able to fully commit to Starwind and Proxmox combination.
Last edited by nick.smith on Tue Feb 17, 2026 9:58 am, edited 1 time in total.
Top
yaroslav (staff)
Staff
Posts: 4309
Joined: Mon Nov 18, 2019 11:11 am

Mon Feb 16, 2026 4:22 pm

Welcome to StarWind Forum and thanks for feedback!

If this setup is TRIAL, NFR, or technical preview license, you can reach out to your account manager to put you in contact with the tech team.
Please update to 20086. 20082 might be unstable.
You can use https://imgur.com/ to share screenshots.
This is the state of the Data nics after a reboot of the Starwind VM, they keep getting marked as Disable network interface.
Please pull the logs and share those with me.
This meant testing we had to revert back to a virtual bridge and use virtIO NIC drivers for Data nics, which meant they were only showing as 1Gbit nics.
This might be a display issue due to drivers used. You can check the bandwidth with iperf.
While following the guide mentioned above it depicts this setup as a 2 node cluster and even shows you how to setup a Qdevice for Proxmox but when it gets to storage Witness is skips over this section, also the Starwind CVM does not currently have this functionality.
Proxmox Quorum is needed for Proxmox cluster. It is not the same as StarWind VSAN witness. StarWind VSAN witness is needed only if the network cards or comms between nodes are not redundant.
For HA devices replicated over NMVe-oF, only Heartbeat is available for now, though.
[Screenshot of CrystalDiskMark ]
That's a tool setting problem. One thread is not enough to max performance out (https://www.starwindsoftware.com/blog/s ... omparison/ and https://www.starwindsoftware.com/nvme-o ... me-cluster).
These numbers seem way too high as a simple data copy in Windows would stall and speeds be stuck at zero for a few moments before picking back up again
File copying is not the same as synthetic performance. File copying is impacted by OS too.
Also, for file copying in Windows, it is better to use robocopy with /j flag.
Top
nick.smith
Posts: 4
Joined: Thu Jan 29, 2026 10:40 am

Tue Feb 17, 2026 12:55 pm

Thanks for the reply Yaroslav

I've uploaded all screenshots to imgur and edited the original post an embedded the screenshots

This is our current trial license
Image

Does this link update the version of CVM?
https://ovf.starwind.com/tmplink/StarWi ... roxmox.zip
or do I need another link to get 20086

I have the support bundle when attempting to use SR-IOV, how should I share them with you? email? they are about 50mb per VM

Here are our iperf out puts when we switched back to using virtual bridge

iperf between SW1 and Proxmox node 2

Code: Select all

infra@SW1:~$ iperf3 -c 172.16.10.2
Connecting to host 172.16.10.2, port 5201
[  5] local 172.16.10.10 port 47388 connected to 172.16.10.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.81 GBytes  24.1 Gbits/sec   69   3.80 MBytes
[  5]   1.00-2.00   sec  2.88 GBytes  24.7 Gbits/sec   41   3.80 MBytes
[  5]   2.00-3.00   sec  2.87 GBytes  24.7 Gbits/sec    0   3.80 MBytes
[  5]   3.00-4.00   sec  2.88 GBytes  24.7 Gbits/sec    1   3.80 MBytes
[  5]   4.00-5.00   sec  2.81 GBytes  24.1 Gbits/sec   40   3.80 MBytes
[  5]   5.00-6.00   sec  2.88 GBytes  24.8 Gbits/sec    0   3.80 MBytes
[  5]   6.00-7.00   sec  2.73 GBytes  23.4 Gbits/sec    1   3.80 MBytes
[  5]   7.00-8.00   sec  2.60 GBytes  22.3 Gbits/sec    0   3.80 MBytes
[  5]   8.00-9.00   sec  2.61 GBytes  22.4 Gbits/sec   44   3.80 MBytes
[  5]   9.00-10.00  sec  2.74 GBytes  23.5 Gbits/sec    0   3.80 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.8 GBytes  23.9 Gbits/sec  196             sender
[  5]   0.00-10.00  sec  27.8 GBytes  23.9 Gbits/sec                  receiver

iperf Done.

perf between SW2 and Proxmox node1

Code: Select all

infra@SW2:~$ iperf3 -c 172.16.10.1
Connecting to host 172.16.10.1, port 5201
[  5] local 172.16.10.20 port 56236 connected to 172.16.10.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.79 GBytes  24.0 Gbits/sec    0   4.02 MBytes
[  5]   1.00-2.00   sec  2.75 GBytes  23.6 Gbits/sec    1   4.02 MBytes
[  5]   2.00-3.00   sec  2.73 GBytes  23.4 Gbits/sec    0   4.02 MBytes
[  5]   3.00-4.00   sec  2.78 GBytes  23.9 Gbits/sec    0   4.02 MBytes
[  5]   4.00-5.00   sec  2.78 GBytes  23.8 Gbits/sec    0   4.02 MBytes
[  5]   5.00-6.00   sec  2.77 GBytes  23.8 Gbits/sec    0   4.02 MBytes
[  5]   6.00-7.00   sec  2.79 GBytes  23.9 Gbits/sec    0   4.02 MBytes
[  5]   7.00-8.00   sec  2.73 GBytes  23.4 Gbits/sec    0   4.02 MBytes
[  5]   8.00-9.00   sec  2.72 GBytes  23.4 Gbits/sec    0   4.02 MBytes
[  5]   9.00-10.00  sec  2.70 GBytes  23.2 Gbits/sec    0   4.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec                  receiver

iperf Done.

interesting is you get slower speeds when communicating with the PVE host that the Starwind VM is hosted on

iperf between SW2 and Proxmox node2

Code: Select all

infra@SW2:~$ iperf3 -c 172.16.10.2
Connecting to host 172.16.10.2, port 5201
[  5] local 172.16.10.20 port 56708 connected to 172.16.10.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.39 GBytes  20.6 Gbits/sec    0   1.77 MBytes
[  5]   1.00-2.00   sec  1.69 GBytes  14.5 Gbits/sec    0   1.77 MBytes
[  5]   2.00-3.00   sec  1.69 GBytes  14.5 Gbits/sec    0   1.89 MBytes
[  5]   3.00-4.00   sec  1.72 GBytes  14.8 Gbits/sec    0   1.89 MBytes
[  5]   4.00-5.00   sec  2.34 GBytes  20.1 Gbits/sec    0   1.98 MBytes
[  5]   5.00-6.00   sec  2.06 GBytes  17.7 Gbits/sec    0   1.98 MBytes
[  5]   6.00-7.00   sec  1.78 GBytes  15.3 Gbits/sec    0   1.98 MBytes
[  5]   7.00-8.00   sec  1.72 GBytes  14.8 Gbits/sec    0   1.98 MBytes
[  5]   8.00-9.00   sec  2.13 GBytes  18.3 Gbits/sec    0   2.11 MBytes
[  5]   9.00-10.00  sec  2.18 GBytes  18.7 Gbits/sec    0   2.11 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  19.7 GBytes  16.9 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  19.7 GBytes  16.9 Gbits/sec                  receiver

iperf Done.
the speeds look correct even tho the Starwind web interface only shows them as 1Gbit nics
only concerning that iSCSI speeds between the CVM and the PVE hosting the VM is slight slower than if you connect to the other PVE node

Let me know if you need any further information from our side

Thanks
Nick
Top
yaroslav (staff)
Staff
Posts: 4309
Joined: Mon Nov 18, 2019 11:11 am

Tue Feb 17, 2026 1:12 pm

Nick,

Networking looks good.
You can update as described here https://knowledgebase.starwindsoftware. ... chine-cvm/.

You can use any file transferring service (e.g., google disk) to transfer logs.
Top
nick.smith
Posts: 4
Joined: Thu Jan 29, 2026 10:40 am

Tue Feb 17, 2026 3:23 pm

tried this in the past to update via the web interface and it says it is already up to date

Image
Top
yaroslav (staff)
Staff
Posts: 4309
Joined: Mon Nov 18, 2019 11:11 am

Tue Feb 17, 2026 4:42 pm

Hi Nick,

Please keep it as is then...
Thanks for your update.
Top
nick.smith
Posts: 4
Joined: Thu Jan 29, 2026 10:40 am

Tue Feb 17, 2026 5:10 pm

Also testing robocopy on the shared starwind storage

Code: Select all

C:\Users\Administrator>robocopy "c:\test-a" "c:\test-b" /j

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Tuesday, February 17, 2026 8:48:51 AM
   Source : c:\test-a\
     Dest : c:\test-b\

    Files : *.*

  Options : *.* /DCOPY:DA /COPY:DAT /J /R:1000000 /W:30

------------------------------------------------------------------------------

                           6    c:\test-a\
100%        New File               1.9 g        2GB - Copy (2).zip
100%        New File               1.9 g        2GB - Copy - Copy.zip
100%        New File               1.9 g        2GB - Copy.zip
100%        New File               1.9 g        2GB.zip
100%        New File               1.9 g        2GBtest - Copy.zip
100%        New File               1.9 g        2GBtest.zip

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         6         6         0         0         0         0
   Bytes :  11.999 g  11.999 g         0         0         0         0
   Times :   0:02:04   0:02:04                       0:00:00   0:00:00


   Speed :           103,830,194 Bytes/sec.
   Speed :             5,941.211 MegaBytes/min.
   Ended : Tuesday, February 17, 2026 8:50:55 AM
It was pretty slow compared to a VM hosted directly on the NVME storage, with it locking up multiple times

Currently the Starwind LUN consist of 2x Virtual disks hosted on each node's LVM which is using a single Dell Express Flash PM1725a 1.6TB SFF (Samsung NVME drives)

Here are the results when the VM is hosted on the storage directly same LVM Dell Express Flash PM1725a 1.6TB SFF not using the Starwind Shared storage

Code: Select all

C:\Users\Administrator>robocopy "c:\test-a" "c:\test-b" /j

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Tuesday, February 17, 2026 8:59:26 AM
   Source : c:\test-a\
     Dest : c:\test-b\

    Files : *.*

  Options : *.* /DCOPY:DA /COPY:DAT /J /R:1000000 /W:30

------------------------------------------------------------------------------

                           6    c:\test-a\
100%        New File               1.9 g        2GB - Copy (2).zip
100%        New File               1.9 g        2GB - Copy - Copy.zip
100%        New File               1.9 g        2GB - Copy.zip
100%        New File               1.9 g        2GB.zip
100%        New File               1.9 g        2GBtest - Copy.zip
100%        New File               1.9 g        2GBtest.zip

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         6         6         0         0         0         0
   Bytes :  11.999 g  11.999 g         0         0         0         0
   Times :   0:00:13   0:00:13                       0:00:00   0:00:00


   Speed :           989,836,682 Bytes/sec.
   Speed :            56,638.910 MegaBytes/min.
   Ended : Tuesday, February 17, 2026 8:59:39 AM

853.85% increase based on these numbers.
Any ideas where I should start looking for bottlenecks?
Top
yaroslav (staff)
Staff
Posts: 4309
Joined: Mon Nov 18, 2019 11:11 am

Tue Feb 17, 2026 5:22 pm

When it comes to tcp, Monitor CPU utilization. Maxing out even a single core will be a sign of a bottleneck.
Top
solemnbaseball
Posts: 1
Joined: Thu May 21, 2026 4:03 am
Contact:
Contact solemnbaseball

Thu May 21, 2026 4:06 am

yaroslav (staff) wrote: ↑
Tue Feb 17, 2026 5:22 pm
 When it comes to tcp, Monitor CPU utilization. Maxing out even a single core will be a sign of a bottleneck.
Thank you for sharing these updates to leverage valuable CPU program updates. Follow up to resolve core load issues effectively.
Top
yaroslav (staff)
Staff
Posts: 4309
Joined: Mon Nov 18, 2019 11:11 am

Thu May 21, 2026 7:16 am

you are welcome
Top
Post Reply

Return to “Early Access”