Dedupe .spspx keeps growing

Public beta (bugs, reports, suggestions, features and requests)

Moderators: art (staff), anton (staff), Anatoly (staff), Max (staff)

jcrowe
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Tue Feb 25, 2014 9:54 pm

I have a deduped 500GB drive with < 200GB used in ESXi, but the space that the .spspx files use is almost 300GB. At this point, not only is the deduplication benefits erased, but now it's using more storage than if it were a non-deduped disk.

The other problem is that these files keep growing, sometimes more than 20GB a day, even though the space used in ESXi is not growing. Should I not be using this for high IO VM's in ESXi?

Also, I am getting thousands of these messages in the log, does this have anything to do with these issues?
2/25 13:56:30.634 bd4 Sw: DDDisk_Dispatch_WriteSameAndUnmap: The current ( non de-duplicated device ) does not support the Unmap command!
2/25 14:03:07.284 fec Sw: DDDisk_Dispatch_WriteSameAndUnmap: The current ( non de-duplicated device ) does not support the Unmap command!


It says that it's a non de-duplicated drive, but when I created it as a deduped drive.
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Wed Feb 26, 2014 9:46 am

V6 or V8 dedupe engine?

We keep all the transactions so unless "delete" is enabled the whole "history" for writes is kept.

With V8 you need to "trim" dedupe volume discarding overwrites and unused snapshots.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
Alex (staff)
Staff
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Wed Feb 26, 2014 2:34 pm

jcrowe,
We have added this functionality. New build with full support for unmap commands will be published today or tomorrow.
Best regards,
Alexey.
eliripoll
Posts: 33
Joined: Sat Dec 08, 2012 8:19 pm
Location: United states

Wed Feb 26, 2014 4:29 pm

HI JCrowe,

Glad your testing to, Sounds like you are seeing the same thing I observed as well, the SPspx file gradually grows even with Idle VM's until its more than the data itself. I also reported this observation as well. you can check out in URL below (which also includes another link from another reported posting)

http://www.starwindsoftware.com/forums/ ... t3352.html

Alex actually was the one working with me, so I believe they are aware and are working on getting it straightened out.

@Alex, Hi you mentioned that it may be published today or tomorrow, does that mean a new key will be provided for beta? current one ends tomorrow.
User avatar
Alex (staff)
Staff
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Wed Feb 26, 2014 4:51 pm

Yes, key will be updated too.
Best regards,
Alexey.
Delo123
Posts: 30
Joined: Wed Feb 12, 2014 5:15 pm

Wed Feb 26, 2014 5:19 pm

So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Wed Feb 26, 2014 8:37 pm

Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.
Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
jcrowe
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Wed Feb 26, 2014 8:51 pm

So would it be a good idea to run something like SDelete in windows to zero out all freed space in the OS?
anton (staff) wrote:Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.
Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Wed Feb 26, 2014 9:12 pm

Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)
jcrowe wrote:So would it be a good idea to run something like SDelete in windows to zero out all freed space in the OS?
anton (staff) wrote:Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.
Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
fbifido
Posts: 125
Joined: Thu Sep 05, 2013 7:33 am

Thu Feb 27, 2014 9:08 pm

anton (staff) wrote:Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)
Can you please post the link ?
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Thu Feb 27, 2014 9:27 pm

The same as it was before:

http://www.starwindsoftware.com/starwind-v8-beta

Re-apply and get the key and the link.
fbifido wrote:
anton (staff) wrote:Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)
Can you please post the link ?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
jcrowe
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Thu Mar 06, 2014 6:43 pm

While the RC release slowed down the growth of the .spspx files, it is adding 5-10 of these files a day. The size used now is over the size of the VM's it is supposed to be depuplicating. Is there a way to force cleanup on these?
User avatar
Alex (staff)
Staff
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Fri Mar 07, 2014 10:25 am

We are working on functionality for executing defragmentation by user command. Also automatic defragmentation scheduler settings will be revisited later.

Please let us know, what kind of tasks are executed on your VMs? Is it SQL database, Web-server or mail server, some specific software?
Best regards,
Alexey.
jcrowe
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Fri Mar 07, 2014 6:28 pm

I'm running a little bit of everything. The highest IO VM's are a MySQL and MSSQL server, following closely by a UTM firewall.
Alex (staff) wrote:We are working on functionality for executing defragmentation by user command. Also automatic defragmentation scheduler settings will be revisited later.

Please let us know, what kind of tasks are executed on your VMs? Is it SQL database, Web-server or mail server, some specific software?
User avatar
Alex (staff)
Staff
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Tue Mar 11, 2014 10:29 am

Thank you!
Best regards,
Alexey.
Post Reply