Everything is broken. Event 10, 20, iScsiPrt

[madmongoose]

Hello, StarWind!

We have been using your software for five years now. Thank You! We stayed on V8 build 11456 (August 08 2017) for a long time, no complaints, everything worked stably, though the synchronization was long. Two days ago, we decided to upgrade to the latest V8 build 13481. The update went smoothly, but problems started a day later. Approximately every 24 hours, the error “Event 10, iScsiPrt. Login request failed. The login response packet is given in the dump data" and "Event 20, iScsiPrt. Connection to the target was lost. The initiator will attempt to retry the connection.". After that, disks from the Windows cluster are disconnected, all virtual machines are in critical state.

Most interestingly, the StarWind service works, I run the powershell script getHAsyncstate - the Synchronization Status: Synchronized state, but in both servers the iscsicpl state is reconnecting. I rebooted the starwind service, also I'm try deleted the targets from iscsicpl and added again - it did not help. Only server reboot works.

Help us please. I will provide all the necessary information.

Yours faithfully!

[yaroslav (staff)]

Hello,

Sorry to read that. Can I have the logs collected with StarWind Log Collector (https://knowledgebase.starwindsoftware. ... collector/) from each node? You can press the Collect diagnostic information button in StarWind Management Console (select the server).

Use Google Drive to share the logs.

[madmongoose]

Thank You for answer!

Our logs. https://drive.google.com/open?id=1evAip ... nF8ONaRJ0A

[yaroslav (staff)]

Hi,

Thank you for waiting so long. There is a new build of StarWind released.
Is the issue still there?
Did you set up CHAP permissions in StarWind VSAN? If yes, that might be the key.
The issue looks to me as iSCSI session hang.

There are iSCSI misconfigurations.
1) Witness has partner connection. In hyperconverged setups (i.e., storage and VMs are running in one box), it is recommended to connect witness disk (called Quorum in your setup) only locally.
2) ais-iscsi-slow target is connected from HV-02 to HV-01 but there is no local connection for ais-iscsi-slow on HV-02. On HV-01, there is a local connection, but no partner connection for ais-iscsi-slow. Please make sure that this target has local and partner connection.

If the issue is still there, please consider re-doing iSCSI initiator part once again. Looks like that the issue is there... Make sure to clean favorite target list before adding new targets.
Whatever-proof procedure is here:
1. Make sure that all targets have local and partner connections ON BOTH NODES.
2. Make sure that StarWind HA devices are synchronized.
3. Move VMs/SQL server out of the node that is to undergo the changes.
4. Pause the node with draining the roles.
5. Go to iSCSI Initiator to do the job.
6. Disconnect targets
7. Clear favorite targets
8. Connect the new targets.
9. Go to Disk management to see if disks are there.
10. Reboot that node.
11. Unpause
12. Repeat to the other node starting from step 1.

I have also noticed that full installation of StarWind VSAN was performed. Here are the commands to run on each server to remove the unnecessary components.
1. Uninstall StarWind SoftwareVSS Provider
Run as administrator: "C:\Program Files\StarWind Software\StarWind\VSS\stop_.bat"

2. Uninstall SMI-S Agent – run commnads below from CMD:
cd "C:\Program Files\StarWind Software\StarWind\OpenPegasus\bin\"
ConfiguratorConsole.exe" --stop --name StarWindSMISAgent
cd"C:\Program Files\StarWind Software\StarWind\OpenPegasus\bin\"
ConfiguratorConsole.exe" --uninstall --name StarWindSMISAgent

3. Uninstall StarWind Cluster service – run commnads below from CMD:
cd C:\Windows\Microsoft.NET\Framework\v4.0.30319
installutil.exe /u "C:\Program Files\StarWind Software\StarWind\StarWindCluster\StarWind.ClusterService.exe"

Also, there are multiple SQL issues in logs... I have 0 expirience in SQL; cannot assist you with that.

[madmongoose]

Hello!

Thanks for the answer.

With the new version the problem is the same. I rolled back to V8 build 11456, it does not have this problem.

There is no password on CHAP and never was.

1. I think the problem is not in quorum, especially on V8 build 11456 everything works.
2. I collected the logs after the crash during the Starwind synchronization, so you considered this a wrong configuration. In normal operation, each node has a local and partner connection.
3. Deleted targets, cleared, added again, rebooted - did not help.

Delete VSS did not work, it gave an error in the screenshot.

As a result, we have version V8 build 11456, it has been working stably for several days, but the speed seems to be lower on it. A few services that I would like to correctly remove:
StarWindNVMeoF
VssHWProviderStarWind

I also collected logs before rolling back to the previous version.
https://drive.google.com/open?id=1evAip ... nF8ONaRJ0A

Thank you!

[yaroslav (staff)]

Hi,

You can remove VSS provider with
cd "C:\Program Files\StarWind Software\StarWind\VSS\"
stop_.bat

Disable StarWind NVMe-oF Target in services. It should be just fine.

I will get to the logs as soon as possible.

[yaroslav (staff)]

Greetings,

I took a closer look at logs. The problem is that the key you currently use is not compatible with the new version. Below, find the events from StarWind VSAN Service logs that support this statement.

5/4 9:49:13.284099 1c08 Srv: iScsiServer::listenConnections: License expired. Refreshing...
5/4 9:49:13.284289 1c08 Srv: iScsiServer::DisconnectAllTargetSessions: Target iqn.2008-08.com.starwindsoftware:10.131.1.61-ais-iscsi-quorum: Stopping session 0x28 from iqn.1991-05.com.microsoft:ais-srv-hv-01.ai-sys.ru,400001370000...
5/4 9:49:13.284314 1c08 C[28], LIN: iScsiConnection::doTransition: Event - SHUTDOWN.

Please log the support case by filling in this form https://www.starwindsoftware.com/support-form. Please mention that your call is related to this thread (just add the link).

[madmongoose]

After change a license key everything is work!

Thank you very much, Starwind Team!

[yaroslav (staff)]

Sounds great!
You are always welcome