How to replace non-failed VSAN storage

Mon Jan 08, 2024 7:36 am

Type [ code ]
text between
[ /code ]
Remove spaces inside brackets to let the magic happen.

wallewek · Tue Jan 09, 2024 11:15 pm

OK, here you go. I'll start with text notes.

Script to force Microsoft Cluster Aware Updates to pause and wait until StarWind nodes are fully synchronized before applying Windows Updates, in case an update requires a host to be rebooted. This is an enhaced and working solution based on the CAU.ps1 example script that is part of StarWindX-provided Powershell collection. You WILL also need to update the script with hostnames.

1. Put the Runner_CAU.PS1 file in folder shared by both hosts, e.g., C:\ClusterStorage\Volume1\CAU\ where Volume1 is a pointer to the Cluster Shared Volume. I created the CAU folder.

2. Go to Microsoft Failover Cluster Manager -> cluster-name -> Configure -> Cluster-Aware Updating > Configure cluster self-updating options -> Advanced Options -> PreUpdateScript

3. Put in the address of the the Runner-CAU.ps1 file

When you run Cluster Aware Updates, it will run the Runner script, which will run the kmCAU.ps1 script which should be in the same folder.
The kmCAU script can be run interactively to check statuses, in which case it will output updates to the screen. It will also log activity to CAU.LOG.

NOTE: the kmCAU.ps1 must be updated with site-specific details, particularly the node names of the servers being updated.

Runner_CAU.ps1

Code: Select all

powershell.exe -sta -file "C:\ClusterStorage\Volume1\CAU\kmCAU.ps1"

kmCAU.ps1

Code: Select all

Import-Module StarWindX
 
# Setting IP addresses of the StarWind storage nodes
$ip1 = "kmhv3" # IP address of node 1
$ip2 = "kmhv4" # IP address of node 2

$logfile = "C:\ClusterStorage\Volume1\CAU\CAU_LOG.txt"
 
# =====================================================================================
# Clear-Content 'C:\ClusterStorage\Volume1\CAU\CAU_LOG.txt'
$time = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R --- Starting StarWind CAU management") -PassThru
# =====================================================================================
 
$server1 = New-SWServer -host $ip1 -user root -password starwind -port 3261
$server2 = New-SWServer -host $ip2 -user root -password starwind -port 3261
 
$server1.Connect()
$server2.Connect()
 
if ($server1.Connected -and $server2.Connected){
 
# =====================================================================================
$time = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R StarWind servers connected") -PassThru
# =====================================================================================
 
do {
 
$result = @()
 
foreach($device1 in ($server1.Devices | where {$_.deviceType -eq 'HA Image'}))
{

$device1.Refresh()
$syncState1 = $device1.GetPropertyValue("ha_synch_status")

# ==================================================================================
$syncPercent = $device1.GetPropertyValue("ha_synch_percent")
$time1 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R $($device1.TargetName) synch state: $($syncState1) ($($syncPercent)%)") -PassThru
# ==================================================================================

 
if ($syncState1 -ne "1")
{
 
# ==================================================================================
# $time1 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R is not synchronized yet. Pausing for 120 seconds") -PassThru
# ==================================================================================
 
$result += $syncState1
 
}
 
}
 
 
foreach($device2 in ($server2.Devices | where {$_.deviceType -eq 'HA Image'}))
{

$device2.Refresh()
$syncState2 = $device2.GetPropertyValue("ha_synch_status")

# ==================================================================================
$syncPercent = $device2.GetPropertyValue("ha_synch_percent")
$time2 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R $($device2.TargetName) synch state: $($syncState2) ($($syncPercent)%)") -PassThru
# ==================================================================================


if ($syncState2 -ne "1") 
{
 
# ==================================================================================
# $time2 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R $($device2.TargetName) is not synchronized yet. Pausing for 120 seconds") -PassThru
# ==================================================================================
 
$result += $syncState2
 
}
 
}
 

# =====================================================================================
 
#Wait for 120 second and rescan until all synched
$result = $result.length

if ($result -gt "0") {Start-Sleep -s 120}
 
} while ($result -gt "0")

} else {
 
# ==================================================================================
$time4 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R StarWind servers not connected. Stopping CAU run. Exiting...") -PassThru
# ==================================================================================
 
# Cancel CAU Process
Stop-CauRun -Force -Wait
 
}
 
$server1.Disconnect()
$server2.Disconnect()

# ==================================================================================
$time5 = (Get-Date -Uformat "%Y/%m/%d %R")
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R Synch OK, StarWind servers disconnected") -PassThru
Add-Content -Path $logfile -Value (Get-Date -Uformat "%Y/%m/%d %R pre-CAU script ended") -PassThru
# ==================================================================================

wallewek · Tue Jan 09, 2024 11:24 pm

Here's another one that is an enhanced and working version of a StarWind-provided example.

I like this one because it makes it much easier and clearer to get detailed synch status. I found other versions less illuminating.

GetHASyncState.ps1

Code: Select all

#
# Get synchronization status of specified HA device and if there is a need (potentially) run synchronization
#

"GetHASYncState -- KW version"

Import-Module StarWindX

while ($true) {

"------------- Running ---------------"

 Write-Host $env:computername, $(Get-Date)

try
{
    #
    # connect to the server
    #
    $server = New-SWServer -host 127.0.0.1 -port 3261 -user root -password starwind

    $server.Connect()

    #
    # Try to find specified device
    #
    $deviceName = "*"
    $partnerTargetName = "*"
    $deviceFound = $false
    
    foreach($device in $server.devices)
    {
            $device.Refresh()
            
            $syncStatus = $device.GetPropertyValue("ha_synch_status")

            Write-Host ""
            Write-Host -NoNewline "DeviceName:" $device.GetPropertyValue("DeviceName") -foreground yellow

            If ( $syncStatus -gt "" )
            {
                $state = $device.GetPropertyValue("state")
                
                Write-Host " -- TargetName:" $device.GetPropertyValue("TargetName") 
                Write-Host -foreground yellow -NoNewline "Device state:" $state 
                
                $stateText = switch ($state)
                {
                    0 { " (Active)" }
                    1 { " (NonActive)" }
                    2 { " (NotLicensed)" }
                    3 { " (Disabled)" }
                    default { "(Undefined)" }
                }

               Write-Host $stateText  -foreground yellow

               Write-Host -NoNewline "Sync Status:" $syncStatus "" -foreground yellow

               $waitForAutoSync = $device.GetPropertyValue("ha_wait_on_autosynch")
                
                if ( $waitForAutoSync -eq "1" )
                {
                    Write-Host "(Waiting for autosynchronization)" -foreground yellow
                }
                else
                {
                    if ( $syncStatus -eq "1" )
                    {
                        #
                        # Device is synchronized. Get synchronization percent and show it
                        #
                        $syncPercent = $device.GetPropertyValue("ha_synch_percent")                        
                        Write-Host "(Synchronized) -- $($syncPercent)%" -foreground yellow

                    }
                    if ( $syncStatus -eq "2" )
                    {
                        #
                        # Device is synchronizing. Get synchronization percent and show it
                        #
                        $syncPercent = $device.GetPropertyValue("ha_synch_percent")
                        Write-Host "(Synchronizing) -- $($syncPercent)%" -foreground yellow
                    }
                    
                    if ( $syncStatus -eq "3" )
                    {
                        #
                        # Device not synchronized. Synchronize current node from partner
                        #
                        Write-Host "Device not synchronized."
                        
                        # Synchronize current node from partner '$($partnerTargetName)'" -foreground yellow

                       # $params = new-object -ComObject StarWindX.Parameters        
                       # $params.AppendParam("deviceID",$device.DeviceId)
                       # $params.AppendParam("partnetTargetName",$partnerTargetName)
                        
                       # $server.ExecuteCommand( 0, "restoreHAPartnerNode", $params)
                        
                        #
                        # If you want to synchronize partners from current node you can comment out code above and uncomment section below
                        #
                        
                        # Device not synchronized. Mark current node as 'Synchronized'. 
                        # WARNING, Command changes Device Status to "Synchronized" without Data Synchronization with HA (High Availability) Partner, 
                        # Device will start processing Client Requests immediately and will be used as Data Synchronization Source for Partner Device.
                        #
                        #Write-Host "Device not synchronized. Mark current node as 'Synchronized'. " -foreground yellow

                        #$params = new-object -ComObject StarWindX.Parameters        
                        #$params.AppendParam("deviceID",$device.DeviceId)
                        
                            #$server.ExecuteCommand( 0, "restoreCurrentHANode", $params)
                        
                       # Start-Sleep -m 5000
                    }
                }
#=======
                Write-Host "ha_partner_nodes_count:" $device.GetPropertyValue("ha_partner_nodes_count")

                $status = $device.GetPropertyValue("ha_partner_node1_sync_status")
                $statusText = switch ($device.GetPropertyValue("ha_partner_node1_sync_status") )
                {
                    1 { " (Synchronized)" }
                    2 { " (Synchronizing)" }
                    3 { " (NOT Synchronized)" }
                    default { "(Undefined)" }
                }
                Write-Host "ha_partner_node1_sync_status: $($status)$($StatusText) --" `
                            $device.GetPropertyValue("ha_partner_node1_sync_percent"), "%"
                Write-Host "ha_partner_node1_is_exist_sync_valid_connection:" `
                            $device.GetPropertyValue("ha_partner_node1_is_exist_sync_valid_connection")
                Write-Host "ha_partner_node1_is_exist_heartbeat_valid_connection:" `
                            $device.GetPropertyValue("ha_partner_node1_is_exist_heartbeat_valid_connection")
                
               # Write-Host "ha_partner_node2_sync_status:" $device.GetPropertyValue("ha_partner_node2_sync_status")
               # Write-Host "ha_partner_node2_sync_percent:" $device.GetPropertyValue("ha_partner_node2_sync_status")




            } 
            else
            {
                write-host -NoNewLine " -- no info"
            }
    }
    
#    if ( $deviceFound -ne $true )
#    {
#        Write-Host "$($deviceName) not found" -foreground red
#    }

}
catch
{
    Write-Host "Exception $($_.Exception.Message)" -foreground red 
}

Write-Host ""

$server.Disconnect( )
pause

}

Wed Jan 10, 2024 12:13 am

Wow!
Thanks for your effort and for sharing the script. Will bookmark this thread

Having synchronization statuses and percentile can be helpful (plus wait time if not sycnhronized).

Thanks a lot!