Troubleshooting write performances

gstar88 · Fri Mar 20, 2026 6:49 am

Hi All,

I have a single 2-node proxmox setup, all NIC 10Gbe and SSD Disks.
I tested network connectivity with iperf and it is 9.9Gbps
I tested raw disk speed and is roughly 400MB/s both sequential reads and writes
However, the vSAN lun, tested inside a windows server with winsat, while giving acceptable sequential reading speeds (180MB/s), has catastrophic write performance (35MB/s). If I stop syncyng wbetween the two nodes is increases to 95MB/s, which is actually still not acceptable.

Any idea where the bottleneck might be?

Thanks!

Fri Mar 20, 2026 8:35 am

Welcome to StarWind Forum!
VM cannot saturate the performance of the entire disk.
Instead, test the performance of that volume in Proxmox. Use FIO and

Code: Select all

fio --name=test --ioengine=libaio --blocksize=4k --rw=randwrite --buffered=0 --iodepth=32 --numjobs=8 --size=50G --runtime=60 --filename=/mnt/volume-name/test.io --group_reporting

Run this test inside the CVM and on Proxmox hosts.

gstar88 · Sat Mar 21, 2026 4:21 am

From inside CVM seems accetable:

root@CVM01:/mnt/sdb1/Volume0/lun0# fio --name=test --ioengine=libaio --blocksize=4k --rw=randwrite --buffered=0 --iodepth=32 --numjobs=8 --size=5G --runtime=60 --filename=/mnt/sdb1/Volume0/test.io --group_reporting
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 8 processes
test: Laying out IO file (1 file / 5120MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=163MiB/s][w=41.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=8): err= 0: pid=121534: Sat Mar 21 04:48:37 2026
write: IOPS=40.1k, BW=157MiB/s (164MB/s)(9406MiB/60010msec); 0 zone resets
slat (usec): min=3, max=5082, avg=59.79, stdev=157.30
clat (usec): min=137, max=77567, avg=6318.23, stdev=4381.74
lat (usec): min=160, max=77574, avg=6378.37, stdev=4360.33
clat percentiles (usec):
| 1.00th=[ 865], 5.00th=[ 1631], 10.00th=[ 2057], 20.00th=[ 2802],
| 30.00th=[ 3589], 40.00th=[ 4621], 50.00th=[ 5473], 60.00th=[ 6259],
| 70.00th=[ 7373], 80.00th=[ 8979], 90.00th=[11600], 95.00th=[14484],
| 99.00th=[21890], 99.50th=[25297], 99.90th=[34341], 99.95th=[39584],
| 99.99th=[54789]
bw ( KiB/s): min=134774, max=175491, per=99.99%, avg=160485.78, stdev=1022.39, samples=960
iops : min=33693, max=43872, avg=40121.17, stdev=255.60, samples=960
lat (usec) : 250=0.01%, 500=0.19%, 750=0.51%, 1000=0.75%
lat (msec) : 2=7.76%, 4=25.13%, 10=50.34%, 20=13.85%, 50=1.45%
lat (msec) : 100=0.02%
cpu : usr=2.09%, sys=5.78%, ctx=1374386, majf=0, minf=96
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2407898,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=157MiB/s (164MB/s), 157MiB/s-157MiB/s (164MB/s-164MB/s), io=9406MiB (9863MB), run=60010-60010msec

Disk stats (read/write):
dm-1: ios=24/2404709, merge=0/0, ticks=176/13110320, in_queue=13110496, util=99.89%, aggrios=24/2408815, aggrmerge=0/551, aggrticks=169/13075041, aggrin_queue=13075218, aggrutil=99.81%
sdb: ios=24/2408815, merge=0/551, ticks=169/13075041, in_queue=13075218, util=99.81%

From Proxmox nearly half speed:

root@proxmox1:/dev/pve# fio --name=test --ioengine=libaio --blocksize=4k --rw=randwrite --buffered=0 --iodepth=32 --numjobs=8 --size=5G --runtime=60 --filename=/dev/CVM/vm-102-disk-2 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.39
Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][w=90.9MiB/s][w=23.3k IOPS][eta 00m:00s]
test: (groupid=0, jobs=8): err= 0: pid=1494168: Sat Mar 21 05:14:41 2026
write: IOPS=24.7k, BW=96.6MiB/s (101MB/s)(5795MiB/60010msec); 0 zone resets
slat (usec): min=3, max=1331, avg=16.82, stdev=17.90
clat (usec): min=943, max=128354, avg=10336.44, stdev=5832.07
lat (usec): min=953, max=128367, avg=10353.26, stdev=5831.90
clat percentiles (usec):
| 1.00th=[ 2180], 5.00th=[ 2999], 10.00th=[ 3752], 20.00th=[ 5211],
| 30.00th=[ 6718], 40.00th=[ 8225], 50.00th=[ 9765], 60.00th=[11338],
| 70.00th=[12911], 80.00th=[14746], 90.00th=[16909], 95.00th=[19006],
| 99.00th=[26870], 99.50th=[34341], 99.90th=[55837], 99.95th=[64226],
| 99.99th=[83362]
bw ( KiB/s): min=86640, max=114136, per=100.00%, avg=98904.47, stdev=831.49, samples=960
iops : min=21658, max=28534, avg=24726.12, stdev=207.88, samples=960
lat (usec) : 1000=0.01%
lat (msec) : 2=0.55%, 4=11.12%, 10=39.84%, 20=44.62%, 50=3.72%
lat (msec) : 100=0.15%, 250=0.01%
cpu : usr=1.37%, sys=6.00%, ctx=1323190, majf=0, minf=1091
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1483567,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=96.6MiB/s (101MB/s), 96.6MiB/s-96.6MiB/s (101MB/s-101MB/s), io=5795MiB (6077MB), run=60010-60010msec

But how can be from windows 30% compared to proxmox, when writing to the vsan disk, and same as proxmox when writing to a direct attached disk?

Sat Mar 21, 2026 11:25 am

Greetings,

Performance inside the VM is dependent on the VM settings more than the storage itself. That's why we normally benchmark on the hypervisor level, because anything going on in the VM is kinda out of scope.
HA devices are 2-way replicas. Effectively, you need to write twice the same block. This creates up to 50% performance cut. What you can try is
1. Stop starwind-virtual-san.service
2. Edit the -ha.swdsk header by increasing the number in the data connection section from <1> to <3>.
3. Start the service.
4. Wait for fast sync to complete.
5. Repeat for another CVM.

Virtualized workloads will always be slower as compared to the storage they are running on by design.
IMO, Hypervisor implies that there are multiple VMs running, that's why it is limiting the performance of an individual device. The performance