2-Node cluster, Starwind storage not available on either host, Help!

[LeoJS]

Hi all,

Sorry for the "Help!" but we are in a pinch. We have a 2-node cluster that has been working fine for months. No hardware issues, but woke this AM and while the two nodes are up the storage is offline and therefore all the customer VMs are down.

Both nodes say unsynchonized. We're at a bit of a loss and seeking a second set of eyes. Any recommendations appreciated, trying to see if there's an ability to purchase per incident support or upgrade to paid to get help.

Thank you

[yaroslav (staff)]

What does the management console look like?
Check the license. If trial, has it expired?
If just not synchronized on both nodes, see this: https://knowledgebase.starwindsoftware. ... -blackout/
For Free version, you can mark the device as synchronized with the script.
1. Open SyncHADevice.ps1,
2. comment $device.Synchronize([SwHaSyncType]::SW_HA_SYNC_FULL, "")
3. uncomment #$device.MarkAsSynchronized()

[LeoJS]

Thank you very much for the response. Here's some more background .. 2-node hyperconverged Hyper-V cluster on StarWind VSAN Free, Windows application, Server [2019/2022]). One HA device (HAImage1, 6.97 TB, synchronous, Heartbeat failover). Working fine until this morning — now both nodes show "Device is not synchronized" / "Partner is not synchronized", so the device won't come online anywhere and the CSV and all VMs are down.

Both hosts are up and the sync channel is passing traffic. All iSCSI logins are rejected:

HA: HASyncNode::registerSession: Unable to register the new client session. The node is not active!
Tgt: *** iScsiTarget::openSession: can't register session. The device 'HAImage1' is not ready.
T[62,1]: ***iScsiTask::startLoginPhase: *ERROR* Login request: device open failed.

Since this is Free (no console management), what is the correct StarWindX/PowerShell procedure to designate the node with the latest data and bring the device online — is SyncHaDevice.ps1 against the chosen node the right approach? And what's the best way to confirm which node has the newest data first?

THANK YOU

[LeoJS]

this was GOLD, thank you:

For Free version, you can mark the device as synchronized with the script.
1. Open SyncHADevice.ps1,
2. comment $device.Synchronize([SwHaSyncType]::SW_HA_SYNC_FULL, "")
3. uncomment #$device.MarkAsSynchronized()

[yaroslav (staff)]

Hope stress is gone
Have a refreshing weekend.

[LeoJS]

As soon as I clean out my pants, I hope to have a refreshing weekend.

Contacted sales to explore a purchase, this was a test to see how well it worked vs. our prior setup just using Hyper-V replication. All seemed great until it wasn't. This customer appears to have frequent power issues, the APC UPS units have been earning their keep but I suspect such outages when batteries deplete / come back up when power restored is not helping.

Thanks again Yaroslav!

[yaroslav (staff)]

You are welcome.
From the VSAN side, you can remove the write-back cache (I have a gut feeling it's there) to make the shared storage less vulnerable to power outages.
https://knowledgebase.starwindsoftware. ... -l1-cache/

[LeoJS]

Great tip, we will explore disabling that write back cache. Thank you.

I think we may have found the root cause of the issue. Recovery via MarkAsSynchronized() in SyncHaDevice.ps1 worked; full sync to the partner is running now. Thanks for that, appreciated very much.

StarWindService.exe crashed twice on the same day (build 8.0.0.20084, Windows Server 2022, 2-node hyperconverged, direct 25GbE sync). First crash at 3:22 PM yesterday triggered a full resync; second crash at 10:32 PM hit the sync-source node mid-resync, which left both nodes not synchronized and storage down.

Crash details: Event 1000 shows faulting module ntdll.dll, exception 0xc0000374 (heap corruption). StarWind's own log shows AsyncTaskCustom::run: WaitForSingleObject(0x0, INFINITE) failed → panicMiniDumpFilter: Unhandled exception: C0000005 → exception inside panicMiniDumpFilter itself.

Should we upgrade to 8.0.0.20089 ASAP? We noticed the release notes list a known issue in build 20084 "sporadic VSAN crash when SCSI UNMAP commands are executed," fixed in 20086. Does our crash look like that issue?

Thank you in advance.

[yaroslav (staff)]

Known issue for 20084.
Yes, update to 20089 (beware of downtime).

[Fonzies50]

Good evening,
On node one, the LUN allows me to view all the information in read-only mode, but on node two, which is not synchronized, it returns a license error.
Thank you for linking me to the discussion I read, but when I run the script (changing the parameters with the IP of the failed node and my image name), it returns "Device not found."

Code: Select all

param($addr="172.22.22.38", $port=3261, $user="root", $password="starwind", $deviceName="havolume1")

Import-Module StarWindX

172.22.22.38 is faulty node

Node 2 Manage error - 2026-08-31 224528.jpg (50.81 KiB) Viewed 7861 times

22.37 is online node

Node 1 Manage(1) - 2026-08-31 224028.jpg (69.6 KiB) Viewed 7861 times

[LeoJS]

Fonzies50 - this thread saved my bacon last month so I'll try to pay some of it forward. Couple things jump out from your screenshots.

Your situation is actually different from mine in one important way. When I had my outage BOTH nodes were unsynchronized and storage was down everywhere - that's the only scenario where the MarkAsSynchronized() edit applies. Your .37 node looks alive and serving (2 sessions, LUN online, and the alert just says the partner isn't synchronized). So I would NOT mark anything as synchronized, especially not on the .38 box - you could end up promoting the stale copy over your good data. You just need to get a normal full sync running again.

On the "Device not found" - the script connected fine, it just doesn't recognize the name. It wants the internal StarWind device name, not the LUN name from the web UI (starwind-vsan01), and it's case sensitive. List what's actually there first:

Code: Select all

Import-Module StarWindX
$server = New-SWServer -host 172.22.22.38 -port 3261 -user root -password starwind
$server.Connect()
$server.Devices | Format-Table Name, TargetName
$server.Disconnect()

Then run SyncHaDevice.ps1 with the exact Name from that output, in its default form (the Synchronize line active, nothing swapped to MarkAsSynchronized) against the node that's out of sync. Might also be worth just trying the Synchronize button in the alert on your .37 node first - that's the safest route either way.

For the license error - my understanding is the Free web UI is monitoring only, so "functionality is not licensed" on Manage LUN is sort of expected. But since your two nodes are behaving differently, compare Settings > License on both. I've seen posts about CVMs losing their license after an update, and I'd guess an unlicensed node won't resync. Also double-check the sync/heartbeat links are passing traffic in both directions - "several unsuccessful synchronization attempts" usually means sync keeps starting and dying, and the .38 node's log should say why.

You're on the CVM/web UI deployment and mine is the Windows application, best to let Yaroslav confirm the specifics but wanted to share in case there are any useful nuggets.

[yaroslav (staff)]

@LeoJS
Thanks for your help!

[Fonzies50]

Good morning and thank you, Leo,
From node 1, when viewing the LUN information, the license appears to be fine, but on node 2, I can't perform any operations. It's as if it were disabled, present but untouchable. I think it's the license's fault, which for some reason was deactivated and blocked my node.
Otherwise:

.\StatusNode1.ps1

Name TargetName
---- ----------
HAImage1 iqn.2008-08.com.starwindsoftware:172.22.22.38-starwind-vsan01
imagefile1 empty

Code: Select all

param($addr="172.22.22.38", $port=3261, $user="root", $password="starwind", $deviceName="iqn.2008-08.com.starwindsoftware:172.22.22.38-starwind-vsan01")

Import-Module StarWindX

try
{
	Enable-SWXLog

	$server = New-SWServer $addr $port $user $password

	$server.Connect()

	$device = Get-Device $server -name $deviceName
	if( !$device )
	{
		Write-Host "Device not found" -foreground red
		return
	}

	if($device.MaintenanceMode -ne [SwHaMaintenanceMode]::Off)
	{
		Write-Host "Device is in maintenance mode" -foreground yellow
	}
	elseif($device.SyncStatus -ne [SwHaSyncStatus]::SW_HA_SYNC_STATUS_SYNC)
	{
		Write-Host "Synchronize device $($device.name)" -foreground yellow
		$device.Synchronize([SwHaSyncType]::SW_HA_SYNC_FULL, "")
		#$device.MarkAsSynchronized()
	}
	else
	{
		Write-Host "Device $($device.name) is synchronized" -foreground yellow
	}
}
catch
{
	Write-Host $_ -foreground red
}
finally
{
	$server.Disconnect()
}

Code: Select all

.\SyncHaDevice.ps1
Device not found

I don't like having these undefined situations because this solution is a test for production deployment. I have nothing on the LUN, and right now, even if one node is fine (the one that has given me less work than the other), the situation isn't stable in my scenario.
I'm changing my strategy. I'll redo the .38, even though it will take longer.

Is this procedure valid for removal?
viewtopic.php?p=32999#p32999

[Fonzies50]

Good morning and thank you, Leo,
From node 1, when viewing the LUN information, the license appears to be fine, but on node 2, I can't perform any operations. It's as if it were disabled, present but untouchable. I think it's the license's fault, which for some reason was deactivated and blocked my node.
Otherwise:

.\StatusNode1.ps1

Name TargetName
---- ----------
HAImage1 iqn.2008-08.com.starwindsoftware:172.22.22.38-starwind-vsan01
imagefile1 empty

Code: Select all

param($addr="172.22.22.38", $port=3261, $user="root", $password="starwind", $deviceName="iqn.2008-08.com.starwindsoftware:172.22.22.38-starwind-vsan01")

Import-Module StarWindX

try
{
	Enable-SWXLog

	$server = New-SWServer $addr $port $user $password

	$server.Connect()

	$device = Get-Device $server -name $deviceName
	if( !$device )
	{
		Write-Host "Device not found" -foreground red
		return
	}

	if($device.MaintenanceMode -ne [SwHaMaintenanceMode]::Off)
	{
		Write-Host "Device is in maintenance mode" -foreground yellow
	}
	elseif($device.SyncStatus -ne [SwHaSyncStatus]::SW_HA_SYNC_STATUS_SYNC)
	{
		Write-Host "Synchronize device $($device.name)" -foreground yellow
		$device.Synchronize([SwHaSyncType]::SW_HA_SYNC_FULL, "")
		#$device.MarkAsSynchronized()
	}
	else
	{
		Write-Host "Device $($device.name) is synchronized" -foreground yellow
	}
}
catch
{
	Write-Host $_ -foreground red
}
finally
{
	$server.Disconnect()
}

.\SyncHaDevice.ps1
Device not found

I don't like having these undefined situations because this solution is a test for production deployment. I have nothing on the LUN, and right now, even if one node is fine (the one that has given me less work than the other), the situation isn't stable in my scenario.
I'm changing my strategy. I'll redo the .38, even though it will take longer.

Is this procedure valid for removal?
viewtopic.php?p=32999#p32999

[yaroslav (staff)]

Trial fits better for testing because it allows UI.
IMO, using Free for testing makes sense only if you do plan to go free straight away.

Is this procedure valid for removal?

Still is.