VSAN CVM 1.7.816.7724 (vSphere 8): node-wide I/O stalls with idle disk - hangedTaskExist, DistCs contention, nodes

rgb · Thu Oct 29, 2026 9:02 pm

Hi all,

We run a 3-node StarWind VSAN Free cluster (Linux CVM appliance) on vSphere 8 in production and have been chasing recurring node evictions for weeks. We ruled out the infrastructure side with measurements; the evidence now points inside the StarWind service. Right now all three HA devices are running on a single copy, because every resync attempt triggers a new stall. Any guidance from the StarWind team is very welcome.

A compact evidence file is attached. The full bundle (raw StarWind logs of four stall windows from all three nodes, per-second disk/CPU samples, thread stacks, ESXi events) is available on request.

Environment

StarWind VSAN CVM v8.0.0 build 20089 / 1.7.816.7724 (Ubuntu 20.04.1, kernel 5.15.0-124), Free license, 3 nodes
vSphere 8 (vCenter 8.0.3, ESXi 25067014 ), 7 ESXi hosts consume the LUNs via software iSCSI (Round Robin iops=1, ALUA SATP rule)
Each CVM: 8 vCPU (reserved), 24 GB RAM (reserved), 4 GB swap, on a 2x EPYC 9654/9754 host
Each CVM: 1x Micron 9300 15.36 TB NVMe in PCIe passthrough -> md0 -> LVM -> XFS (13.96 TiB)
3 HA devices, 2-way sync replication, thick, 6.5 TiB each: SW-0102 (cvm-01/02), SW-0103 (cvm-01/03), SW-0203 (cvm-02/03)
Created with StarWindX: FailoverStrategy=0 (heartbeat on management network), SyncSessionCount=1, ALUAOptimized=true, CacheMode=none, BitmapStoreType=1, BitmapStrategy=1, MaxUnmap=128, SectorSize=512
Network: 2x 10GbE per host (vDS), dedicated VLANs for iSCSI-A, iSCSI-B and sync (MTU 9000)
Workload: 109 VMs on 3 VMFS6 datastores, Veeam backups (snapshots + hot-add), vSphere Replication (RPO 10 min) on 17 VMs

Symptom
Several times per day a node is evicted by its partner:

Code: Select all

HA: HANode::onScsiOp_CustomControl_HeartbeatPingShutdown: Event: 'not restorable' flag is 0, shutdown reason 'timeout on local underlying storage device', async update header flag is 1.

16 evictions on 10/29 before 17:00. ESXi sees VMFS heartbeat timeouts (1,051 in 22 h, all on the StarWind datastores), APD and path churn. After an eviction, the resync either aborts or completes and the node is evicted again a few minutes later (e.g. SW-0102 resync completed 17:24:39, evicted 17:27:52).

Best capture so far: 10/29 17:17, node cvm-03
cvm-02 was receiving two resyncs (SW-0102 from cvm-01, SW-0203 from cvm-03). SW-0103 was fully synchronized between cvm-01 and cvm-03 and not involved in any sync.

1) The first symptom on cvm-03 is mirrored writes to the resyncing partner (cvm-02) not completing:

Code: Select all

17:17:29 HA: Ssc_Request_Task::complete: Warning(Partner send): ... GeneralOpCode(0x8A) ... PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 3676 ms ... Current target: '...53-sw-0203-n01-n03', Partner target: '...52-sw-0203-n01-n02'!

These waits grow up to 85.8 seconds. cvm-02 at that moment had StarWindService pinned at about one core with light disk activity.

2) From 17:17:29 to about 17:18:45 the NVMe of cvm-03 is completely idle (0-8 completed ops per 2 s, nothing in flight), while StarWindService on cvm-03 holds about one core. The management port 3261 keeps answering. dmesg is clean.

3) The hot threads run in user space and wait on a futex - no kernel time:

Code: Select all

17:17:35 TID=3660940 user=158 sys=0 (ticks/2s, 200 = 1 core) wchan=futex_wait_queue_me
17:17:46 TID=3379676 user=226 sys=0 wchan=0
stack: futex_wait_queue_me <- futex_wait <- do_futex <- __x64_sys_futex

4) The stall is node-wide. The healthy device on the same node hangs too:

Code: Select all

17:17:48 Common: CStarWindStorageDevice::hangedTaskExist: Storage device hanged operation: location = C:\starwind\storage\mnt\md0\vol-nvme01\SW-0103-N01.img, ... cdb[0] = 0x8A / 0x88

followed by DistCs::enter retries (CSid = 0xCCC5), and at 17:18:44 SW-0103 on cvm-03 changes its own sync status to 3 (not synchronized). All 7 ESXi hosts lost the VMFS heartbeat on SW-0203 and SW-0103 for 60-90 seconds.

Our reading: during a resync, a partner that is busy (one thread pinned) responds slowly to mirrored writes, the source node blocks on those partner sends, and that blocks I/O for every device on the source node, including devices that are not involved.

Earlier the same day we saw the same pattern, with DistCs contention and evictions:

Code: Select all

15:53 cvm-02: DistCs::leaveOnTheDistributeSharersSide: (CSid = 0xCCC5) LEAVE command on dist sharer(Id = 0xD0) can't be complete. Sharer return BUSY.
15:54 cvm-02: DistCs::forceLeaveOnTheDistributeSharersSide: *WARNING* (CSid = 0xCCC5) FORCE LEAVE command on dist sharer(Id = 0xD0) refused.
16:43:44 cvm-02 SW-0203 resync completes -> 16:43:46 DistCs retries start -> 0x89 takes 11.8 s -> evicted 16:44:18

What we ruled out (with measurements)

Memory: 24 GB reserved, about 50% used, swap 0 (an earlier incident with 16 GB and swap exhaustion was fixed).
CPU contention: CVM CPU reserved, ESXi hosts at 7-13% CPU.
VM snapshots on the CVMs: removed.
NVMe / kernel / XFS: disk idle during stalls, SMART PASSED, dmesg clean, image files contiguous (2,363-5,553 extents per 6.5 TiB image), hot threads with sys=0.
Network: sync VLAN jumbo ping OK, under 1 Gbit/s at event times.
VMFS ATS heartbeat: disabled on all hosts per VMware KB 2113956. It reduced heartbeat timeouts, but the 17:17 stall happened with it disabled.

Questions

Is this a known issue ? Is there a newer build that fixes it?
Why does a slow partner during resync block all I/O on the source node, including a device that is not part of that resync? Can partner-send waits be bounded or isolated per device?
Can resync traffic or priority be throttled so a resync does not saturate the target node?
What does DistCs CSid 0xCCC5 protect, and what does "FORCE LEAVE ... refused" mean for the HA device?
Recommended settings for ~100 VMs / 7 hosts per LUN: SyncSessionCount, number of heartbeat channels, CVM vCPU sizing (the hot thread caps at one core), ESXi iSCSI settings.
How do we safely bring the three devices back to two copies without triggering the stall again (resync one device at a time, maintenance mode, a specific procedure)?
What else should we collect (debug log level, support bundle, a dump of the hot thread)?

Thank you.

Code: Select all

StarWind VSAN CVM v8.0.0 build 20089 - KEY EVIDENCE - 2026-10-29 (local time UTC-3)
3 CVMs (8 vCPU reserved, 24 GB reserved, Micron 9300 NVMe passthrough, XFS), 3 HA devices 2-way,
7 ESXi 8 hosts via iSCSI, 109 VMs. Nodes: cvm-01 .51, cvm-02 .52, cvm-03 .53.
Devices: SW-0102 (cvm-01/cvm-02), SW-0103 (cvm-01/cvm-03), SW-0203 (cvm-02/cvm-03).

STATE AT 17:17: cvm-02 was receiving TWO resyncs (SW-0102 from cvm-01, SW-0203 from cvm-03).
SW-0103 was SYNCHRONIZED on cvm-01 and cvm-03 (no sync involved). ESXi ATS heartbeat already disabled.

[E1] cvm-03: first symptom = mirrored writes to the RESYNCING partner (cvm-02) not completing
17:17:29 HA: Ssc_Request_Task::complete: Warning(Partner send): ... GeneralOpCode(0x8A) buf_size(4096),
  PartnerOpCode(0xF2), PartnerDiffTimeCompleteEXEC = 3676 ms ... Current target: '...53-sw-0203-n01-n03',
  Partner target: '...52-sw-0203-n01-n02'!
Partner-send warnings on cvm-03, 17:17:20-17:19:10, by partner (count per 10 s):
  to cvm-02 (SW-0203, resync target): 17:17:2x 25 | 3x 61 | 4x 17 | 18:0x 5 | 1x 10 | 2x 7 | 3x 18 | 4x 88 | 5x 61  max 85,836 ms
  to cvm-01 (SW-0103, healthy):       17:17:3x 6  | 4x 24 | 5x 3  | 18:0x 2 | 4x 32                        max 70,416 ms
cvm-01 was healthy at that time (disk busy 15-35k ops/2s, service 6-40% CPU), so the SW-0103 waits are
cvm-03 itself being stalled.

[E2] cvm-03: disk IDLE for ~75 s while StarWindService holds ~1 core (2-s samples, /proc/diskstats)
17:17:20 nvme ops+=14016 inflight=0 busy+=1360ms | sw=18.8% wine=6.2% | 3261=OK
17:17:24 nvme ops+=7601 inflight=0 busy+=624ms | sw=62.5% wine=6.2% | 3261=OK
17:17:29 nvme ops+=276 inflight=0 busy+=120ms | sw=106.2% wine=0% | 3261=OK
17:17:33 nvme ops+=8457 inflight=0 busy+=312ms | sw=100% wine=0% | 3261=OK
17:17:38 nvme ops+=64 inflight=0 busy+=76ms | sw=100% wine=0% | 3261=OK
17:17:42 nvme ops+=43 inflight=0 busy+=68ms | sw=100% wine=0% | 3261=OK
17:17:47 nvme ops+=73 inflight=0 busy+=16ms | sw=100% wine=0% | 3261=OK
17:17:52 nvme ops+=2 inflight=0 busy+=8ms | sw=94.1% wine=0% | 3261=OK
17:17:56 nvme ops+=0 inflight=0 busy+=0ms | sw=100% wine=0% | 3261=OK
17:18:01 nvme ops+=1 inflight=0 busy+=4ms | sw=100% wine=0% | 3261=OK
17:18:06 nvme ops+=1 inflight=0 busy+=4ms | sw=112.5% wine=0% | 3261=OK
17:18:20 nvme ops+=4 inflight=0 busy+=24ms | sw=112.5% wine=0% | 3261=OK
17:18:25 nvme ops+=4 inflight=0 busy+=16ms | sw=100% wine=0% | 3261=OK
17:18:29 nvme ops+=3 inflight=0 busy+=12ms | sw=100% wine=0% | 3261=OK
17:18:44 nvme ops+=245 inflight=0 busy+=84ms | sw=112.5% wine=0% | 3261=OK
17:18:48 nvme ops+=105 inflight=0 busy+=100ms | sw=100% wine=0% | 3261=OK
17:18:53 nvme ops+=696 inflight=0 busy+=372ms | sw=100% wine=0% | 3261=OK
17:18:57 nvme ops+=2018 inflight=0 busy+=476ms | sw=106.7% wine=0% | 3261=OK
  (port 3261 answered on every sample; dmesg clean; no in-flight I/O on the NVMe)

[E3] cvm-03 hot thread during the stall (per-thread utime/stime delta, 200 ticks/2 s = 1 core)
17:17:35 TID=3660940 user=158 sys=0 wchan=futex_wait_queue_me
17:17:37 TID=3379676 user=174 sys=0 wchan=futex_wait_queue_me
17:17:43 TID=3660940 user=209 sys=0 wchan=futex_wait_queue_me
17:17:46 TID=3379676 user=226 sys=0 wchan=0
  stack: futex_wait_queue_me <- futex_wait <- do_futex <- __x64_sys_futex   (user space spin + futex)

[E4] cvm-03: the HEALTHY device hangs too (node-wide stall)
17:17:48 Common: CStarWindStorageDevice::hangedTaskExist: Storage device hanged operation:
  location = C:\starwind\storage\mnt\md0\vol-nvme01\SW-0103-N01.img, cdb[0] = 0x8A / 0x88
hangedTaskExist per 10 s: 17:17:4x 774 | 5x 641 | 18:0x 1892 | 1x 2731 | 2x 2256 | 3x 5499 | 4x 13489
DistCs::enter (CSid 0xCCC5) retries per 10 s: 17:18:0x 168 | 1x 176 | 2x 200 | 3x 204 | 4x 133
17:17:56 cvm-02: SW-0203 resync aborts (status 3 from 2)
17:18:44 cvm-03: SW-0103 "changed own sync status to 3 from 1" (healthy device lost its 2nd copy)

[E4b] cvm-02 (the resync TARGET) at the same time: StarWindService ~1 core, NVMe lightly used
17:17:20 nvme ops+=4135 inflight=0 busy+=864ms | sw=125% wine=0% | 3261=OK
17:17:25 nvme ops+=941 inflight=0 busy+=500ms | sw=93.8% wine=0% | 3261=OK
17:17:29 nvme ops+=1810 inflight=0 busy+=772ms | sw=106.7% wine=0% | 3261=OK
17:17:42 nvme ops+=14656 inflight=0 busy+=1184ms | sw=12.5% wine=0% | 3261=OK
17:17:47 nvme ops+=13630 inflight=0 busy+=892ms | sw=113.3% wine=0% | 3261=OK
17:18:00 nvme ops+=4151 inflight=0 busy+=416ms | sw=125% wine=0% | 3261=OK
17:18:04 nvme ops+=5370 inflight=0 busy+=576ms | sw=106.2% wine=0% | 3261=OK
17:18:09 nvme ops+=20483 inflight=0 busy+=540ms | sw=75% wine=0% | 3261=OK
17:18:52 nvme ops+=338 inflight=0 busy+=380ms | sw=87.5% wine=0% | 3261=OK
17:18:57 nvme ops+=418 inflight=0 busy+=364ms | sw=106.2% wine=0% | 3261=OK

[E5] ESXi (7 hosts, VMFS heartbeat as plain SCSI read/write since ATS-HB was disabled)
17:17:40-17:17:45 heartbeat timedout on SW-0203 (6 hosts); 17:17:44-17:17:48 on SW-0103 (7 hosts)
17:18:49-17:19:12 recovered  => 60-90 s with no I/O completing on both LUNs served by cvm-03

[E6] Same pattern earlier the same day (raw logs available)
15:53 cvm-02: 239 DistCs retries, "LEAVE ... Sharer return BUSY", "TIMEOUT_NOTIFY",
      "FORCE LEAVE command on dist sharer refused", FileIO up to 26,576 ms -> evicted 15:54:13
16:43:44 cvm-02 SW-0203 resync completes; 16:43:46 DistCs retries start; 0x89 takes 11,788 ms;
      evicted 16:44:18 ("timeout on local underlying storage device")
17:24:39 cvm-02 SW-0102 resync completes -> evicted again 17:27:52
16 evictions across the 3 nodes on 10/29 before 17:00.

RULED OUT: memory (24 GB, swap 0), CPU contention (reserved, hosts 7-13%), VM snapshots (removed),
NVMe/kernel (idle disk, SMART OK, dmesg clean), XFS (images contiguous, 2,363-5,553 extents per
6.5 TiB), network (jumbo ping OK, <1 Gbit/s), ESXi ATS heartbeat (disabled - stall reproduced).

Fri Oct 30, 2026 8:48 am

Welcome to the StarWind forum. Brace yourself for a loooooooooooong text

I'd like to hear more about the networking. VLANs are not supported. You need to use dedicated networking for iSCSI and SYNC. It should be a dedicated port on the server. Are the servers connected directly, or are they running through the switch?

Several times per day a node is evicted by its partner:

Correct me if I am wrong: are you referring to the replication drop in this thread?

Please share here https://www.starwindsoftware.com/support-form (your reference 1568041)
- StarWind CVM host Support Bundles
- StarWind CVM support bundles
- Network diagram (I need to see how sync and iSCSI are running).
- Network Card manufacturer

Side note:
From what I read the issue seems to be related to networking or running out of descriptors. To rule out the former, try setting MTU to 1500 for the entire network stack (incl. client and storage side (CVM and ESXi) iSCSI network, and SYNC).
Again, no VLANs, pls.

responds slowly to mirrored writes,
Can resync traffic or priority be throttled so a resync does not saturate the target node?

Try:
- tweaking synchronization priority to 5% (script or inside the config files).
- https://knowledgebase.starwindsoftware. ... xpectedly/.

Now let me answer your questions:

Is this a known issue ? Is there a newer build that fixes it?

Nope. Looks like something is funny with this system. That's why I need a network diagram and logs.

Why does a slow partner during resync block all I/O on the source node, including a device that is not part of that resync? Can partner-send waits be bounded or isolated per device?

During resync, both resync and client requests hammer the storage of the active node. The non-active node does nothing but catch up with the partner.
What you can do is reduce synchronization priority so that sync hammers storage not that vigorously.

What does DistCs CSid 0xCCC5 protect, and what does "FORCE LEAVE ... refused" mean for the HA device?

I need context (i.e., logs).

Recommended settings for ~100 VMs / 7 hosts per LUN: SyncSessionCount, number of heartbeat channels, CVM vCPU sizing (the hot thread caps at one core), ESXi iSCSI settings.

No recommendations on this matter. See best practices here https://www.starwindsoftware.com/best-p ... practices/.

How do we safely bring the three devices back to two copies without triggering the stall again (resync one device at a time, maintenance mode, a specific procedure)?

Does "stall" refer to downtime?
If you plan to run a 2-way mirror, you can do it simply by removing the replication partner.
As a side note, I'd like to share this topology: https://www.starwindsoftware.com/resour ... hitecture/

rgb · Fri Oct 30, 2026 3:42 pm

Thanks for the quick reply. I will upload the CVM support bundle, the network diagram and the NIC details through the support form with reference 1568041.

A few clarifications, because I think my first post was not clear enough:

1) What we mean by eviction and stall
- Eviction = replication drop: the HA device on one node goes to "Not synchronized" with shutdown reason 'timeout on local underlying storage device'.
- Stall = a 60-90 second period on one CVM during which NO I/O completes on ANY HA device served by that node, including devices that are not part of any resync. All 7 ESXi hosts log VMFS heartbeat timeouts on those LUNs and the VMs freeze. No data loss and no full outage, but production freezes.

2) The storage is NOT being hammered during the stall
Per-2-second samples of /proc/diskstats on the active node (cvm-03, 10/29):
17:17:24 nvme ops+=7601 busy+=624ms StarWindService=62%
17:17:42 nvme ops+=43 busy+=68ms StarWindService=100%
17:17:56 nvme ops+=0 busy+=0ms StarWindService=100%
17:18:29 nvme ops+=3 busy+=12ms StarWindService=100%
The NVMe sits idle with nothing in flight for over a minute, while StarWindService holds one core in user space (sys time = 0) and its threads wait on a futex. Network traffic was under 1 Gbit/s during the same window.

3) The service degrades over time and a restart clears it
- StarWindService RSS: 0.44 GB on cvm-02 55 minutes after a restart, versus 11.1 GB (cvm-01) and 10.9 GB (cvm-03) after 13 days 21 hours. Same configuration, 2 HA devices per node, no L1 cache. Roughly 0.8 GB per day.
- Threads (177-214) and open file descriptors (914-1016, limit 65536) are stable, so this is not descriptor exhaustion.
- The problems start about two weeks after each restart: deployed Sep 7 -> first drops Sep 25 -> crisis Oct 1; restarted Oct 1 -> crisis Oct 15; restarted Oct 15 -> drops again from Oct 27, crisis Oct 29.
- Right after restarting cvm-02, the fast sync of SW-0102 completed in 46 seconds. The same resync had aborted repeatedly before the restart.
- All three CVMs were restarted on 10/29 between 17:53 and 18:59. Since then there have been no drops and no stalls. We will track StarWindService RSS over the coming days and share the growth curve.
Is ~11 GB RSS after two weeks expected for this configuration?

4) Networking (details in the attached diagram)
Each CVM host has 2x 10GbE uplinks (Intel E810-C on two hosts, Intel X540-AT2 on the third) on one vDS, connected to two Arista DCS-7050TX-64 switches with a 40 Gbps inter-switch link. Uplink 1 carries general traffic and iSCSI-A; Uplink 2 carries iSCSI-B and sync, so sync between the three CVMs stays on the same switch. iSCSI-A, iSCSI-B and sync are separate VLANs with MTU 9000 end-to-end; there are no direct-attach links. The full vDS configuration (teaming, traffic shaping, NIOC) is in the diagram.
Given that the disk is idle, the network is lightly used during the stalls, and a service restart clears the condition, we would like to focus on the service first.

Our layout already follows the grid model you linked: three 2-way devices on pairs 01-02, 01-03 and 02-03.

Thanks.