Tape Redirector Locking Up

Tape drive and auto-loader redirector over iSCSI

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
solaratm
Posts: 4
Joined: Mon Nov 16, 2015 7:14 pm

Mon Nov 16, 2015 7:23 pm

I have been using the tape redirector/iscsi san free version for a little over a year on hyperv server 2012 r2 core to redirect a tape drive to a guest. Everything was working fine until recently. Now about once a week, our nightly guest backup fails due to the tape not being available. Looking on the host, the starwind process is using over 3GB of memory. Trying to restart the service or end the process doesn't work. We have to restart the host and then the service starts working as it should. Nothing has changed on the host at all during this time period (no updates, no software added, no new hardware) I updated to the latest available version and the problem still occurs. Any idea what the issue may be?
Thanks
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Tue Nov 17, 2015 2:05 pm

Tape is not cached so if memory usage increases over time it looks like a memory leak. Can you monitor with PerfMon or Task Manager a) memory on process started on b) when lock-up actually happens?
Also please check do you have exclusive open of your tape with StarWind so no other process can interfere with what we do. Did you install anything new on your host?
solaratm wrote:I have been using the tape redirector/iscsi san free version for a little over a year on hyperv server 2012 r2 core to redirect a tape drive to a guest. Everything was working fine until recently. Now about once a week, our nightly guest backup fails due to the tape not being available. Looking on the host, the starwind process is using over 3GB of memory. Trying to restart the service or end the process doesn't work. We have to restart the host and then the service starts working as it should. Nothing has changed on the host at all during this time period (no updates, no software added, no new hardware) I updated to the latest available version and the problem still occurs. Any idea what the issue may be?
Thanks
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
solaratm
Posts: 4
Joined: Mon Nov 16, 2015 7:14 pm

Wed Nov 18, 2015 1:57 pm

The amount when it locks up seems to very. This morning it was 1016mb. But I may have jumped the gun saying locked up. The service still responds to the management console, but the iscsi initiator on the guest system can no longer connect. Looking in the log, after the last restart, this is repeated multiple times after 1 successful backup on tape then the next backup failing.



11/18 8:32:14.390 a08 Srv: Accepted iSCSI connection from 100.100.1.2:57147 to 100.100.1.1:3260. (Id = 0xea2)
11/18 8:32:14.390 a08 C[ea2], FREE: Event - CONNECTED.
11/18 8:32:14.390 a08 C[ea2], XPT_UP: T3.
11/18 8:32:14.390 b4b8 C[ea2], XPT_UP: Login request: ISID 0x400001370001, TSIH 0x0000.
11/18 8:32:14.390 b4b8 C[ea2], XPT_UP: Event - LOGIN.
11/18 8:32:14.390 b4b8 C[ea2], IN_LOGIN: T4.
11/18 8:32:14.390 b4b8 Params: <<< String param 'InitiatorName': received 'iqn.1991-05.com.microsoft:msi03.solaratm.local', accepted 'iqn.1991-05.com.microsoft:msi03.solaratm.local'
11/18 8:32:14.390 b4b8 Params: <<< Enum param 'SessionType': received 'Normal', accepted 'Normal'
11/18 8:32:14.390 b4b8 Params: <<< String param 'TargetName': received 'iqn.2008-08.com.starwindsoftware:vh103-tape', accepted 'iqn.2008-08.com.starwindsoftware:vh103-tape'
11/18 8:32:14.390 b4b8 Params: <<< Enum param 'AuthMethod': received 'None', accepted 'None'
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Thu Nov 19, 2015 3:15 pm

Please don't embed any logs in posts as they are getting truncated :)

https://forums.starwindsoftware.com/vie ... ?f=5&t=740

Just get full log, zip and send to support@starwind.com with your comments and please answer other questions I've asked. Thanks!
solaratm wrote:The amount when it locks up seems to very. This morning it was 1016mb. But I may have jumped the gun saying locked up. The service still responds to the management console, but the iscsi initiator on the guest system can no longer connect. Looking in the log, after the last restart, this is repeated multiple times after 1 successful backup on tape then the next backup failing.



11/18 8:32:14.390 a08 Srv: Accepted iSCSI connection from 100.100.1.2:57147 to 100.100.1.1:3260. (Id = 0xea2)
11/18 8:32:14.390 a08 C[ea2], FREE: Event - CONNECTED.
11/18 8:32:14.390 a08 C[ea2], XPT_UP: T3.
11/18 8:32:14.390 b4b8 C[ea2], XPT_UP: Login request: ISID 0x400001370001, TSIH 0x0000.
11/18 8:32:14.390 b4b8 C[ea2], XPT_UP: Event - LOGIN.
11/18 8:32:14.390 b4b8 C[ea2], IN_LOGIN: T4.
11/18 8:32:14.390 b4b8 Params: <<< String param 'InitiatorName': received 'iqn.1991-05.com.microsoft:msi03.solaratm.local', accepted 'iqn.1991-05.com.microsoft:msi03.solaratm.local'
11/18 8:32:14.390 b4b8 Params: <<< Enum param 'SessionType': received 'Normal', accepted 'Normal'
11/18 8:32:14.390 b4b8 Params: <<< String param 'TargetName': received 'iqn.2008-08.com.starwindsoftware:vh103-tape', accepted 'iqn.2008-08.com.starwindsoftware:vh103-tape'
11/18 8:32:14.390 b4b8 Params: <<< Enum param 'AuthMethod': received 'None', accepted 'None'
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
solaratm
Posts: 4
Joined: Mon Nov 16, 2015 7:14 pm

Thu Nov 19, 2015 3:26 pm

The log isn't truncated. I just copied one section of it that is repeated hundreds of times. Nothing new was installed to the host. Windows updates are turned off and it is only hosting a single guest. Being as it is a hyper v core install, nothing else can use the tape because there is no software other than the hypervisor running. I wasn't able to restart the server yet, but the memory is now up to 1739mb. As soon as I can restart, I will post that number. And I will send a full log.
Thanks
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Thu Nov 19, 2015 4:01 pm

OK, I'm brining in engineers to take a look!
solaratm wrote:The log isn't truncated. I just copied one section of it that is repeated hundreds of times. Nothing new was installed to the host. Windows updates are turned off and it is only hosting a single guest. Being as it is a hyper v core install, nothing else can use the tape because there is no software other than the hypervisor running. I wasn't able to restart the server yet, but the memory is now up to 1739mb. As soon as I can restart, I will post that number. And I will send a full log.
Thanks
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
solaratm
Posts: 4
Joined: Mon Nov 16, 2015 7:14 pm

Mon Nov 23, 2015 3:31 pm

After reboot and one backup using the tape, the process uses 3.5mb.
Vladislav (Staff)
Staff
Posts: 180
Joined: Fri Feb 27, 2015 4:31 pm

Fri Nov 27, 2015 4:03 pm

Hello,

Quick community update:

Our R&D Department is investigating this issue at the moment.
Vladislav (Staff)
Staff
Posts: 180
Joined: Fri Feb 27, 2015 4:31 pm

Wed Dec 16, 2015 9:07 am

Hello solaratm,

Unfortunately, we did not succeed in reproducing the issue on current version of StarWind.

Could you please upgrade your StarWind to the latest build and try to reproduce it.

Thank you.
Post Reply