Starwind service can't be stopped

CedricT · Thu Sep 09, 2021 6:54 am

Hi all,

We had a strange issue on cluster 2 nodes Windows. Everything was synched but when we tried to restart one node, the starwind service couldn't be stopped. So we had to kill the process manually.

Here what we find in the log repeatly:

9/8 16:01:53.446587 1328 HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:53.446637 ec HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:53.446647 1cbc HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:53.446656 149c HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:56.868243 14e8 HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:57.055061 1ebc Srv: StarWindService::doRun: Waiting for 5 seconds for clean up to complete (6 refs)...
9/8 16:01:57.555780 da4 HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:57.555860 f8c HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:57.555877 10d8 HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:59.445986 a94 ntsrv: *** NtService::watchdogProc: watchdog: service is still stopping... Updating service status.
9/8 16:01:59.447042 1cbc HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:59.447100 ec HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:59.447110 149c HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:01:59.447119 1328 HA: HAPartneriScsiChannel::permanentConnectionPollingThread: Waiting 6 sec before retry sending backward connection command ...
9/8 16:02:02.055388 1ebc Srv: StarWindService::doRun: Waiting for 5 seconds for clean up to complete (6 refs)...

Thu Sep 09, 2021 7:17 am

Cedric,

Thanks for contacting us. Were you able to collect the minidump before termination? In this situation, minidump carries all the info. Also, I will need logs from 2 servers and when roughly the incident happened. Use log collector to collect the logs https://knowledgebase.starwindsoftware. ... collector/

CedricT · Thu Sep 09, 2021 9:08 am

Hi,

Just sent the logs.Problem happened yesterday around 3pm43. There is no minidump because windows didn't crash since we successfully killed the starwind process and OS has been able to complete a reboot normally after. Important thing we had a full sync started after the reboot (but i think that's normal).

Thanks for your help.

CedricT · Mon Sep 13, 2021 1:16 pm

Hi,

Did you get the logs?

Sincerely,

Mon Sep 13, 2021 2:28 pm

No. Can you PM them please?

CedricT · Tue Sep 14, 2021 12:01 pm

Hi,

We sent them to support@starwind.com.

Sincerely,

Tue Sep 14, 2021 12:26 pm

Now I have the logs. Will keep you posted.

Thu Sep 16, 2021 12:55 pm

Cedric,

This behavior is to be fixed by the build to come.
Are you using L2 cache? If so, for now, please try removing L1 (see here https://knowledgebase.starwindsoftware. ... -l1-cache/) and L2 cache (see here https://knowledgebase.starwindsoftware. ... dance/661/). See more about StarWind VSAN caching https://knowledgebase.starwindsoftware. ... rinciples/.
I'd recommend using a separate HA device instead of an L2 cache just to boost both reads and writes.

CedricT · Thu Aug 10, 2023 9:13 am

Hi,

We encountered the problem again. Using the version 15020.

10:49 AM

8/9 4:05:23.009532 10c0 Srv: StarWindService::dorun: Waiting for 5 seconds for clean up to complete (8 refs)...
8/9 4:05:24.869930 2478 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:24.947690 1fe4 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:24.947709 21f0 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:24.947732 1d8c HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:25.635204 1f4c HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:25.635265 1f08 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:25.635284 1f78 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:25.635302 1e50 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:28.025122 10c0 Srv: StarWindService::dorun: Waiting for 5 seconds for clean up to complete (8 refs)...
8/9 4:05:30.885144 2478 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:30.963607 1fe4 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:30.963663 1d8c HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:30.964032 21f0 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:31.650968 1f08 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:31.651082 1f78 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:31.651116 1f4c HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:31.651134 1e50 HA: HAPartneriScsiChannel::permanentconnectionpollingthread: Waiting 6 sec before retry sending backward connection command ...
8/9 4:05:33.025296 10c0 Srv: StarWindService::dorun: Waiting for 5 seconds for clean up to complete (8 refs)...
8/9 4:05:34.493870 2688 ntsrv: *** NtService::watchdogproc: watchdog: service is still stopping... Updating service status.

Thu Aug 10, 2023 10:31 am

Please update to 15159.