StarWind iSCSI SAN
StarWind Native SAN for Hyper-V
 

Dedupe .spspx keeps growing

Public beta (bugs, reports, suggestions, features and requests)

Moderators: art (staff), anton (staff), Anatoly (staff), Max (staff)

Dedupe .spspx keeps growing

Postby jcrowe » Tue Feb 25, 2014 9:54 pm

I have a deduped 500GB drive with < 200GB used in ESXi, but the space that the .spspx files use is almost 300GB. At this point, not only is the deduplication benefits erased, but now it's using more storage than if it were a non-deduped disk.

The other problem is that these files keep growing, sometimes more than 20GB a day, even though the space used in ESXi is not growing. Should I not be using this for high IO VM's in ESXi?

Also, I am getting thousands of these messages in the log, does this have anything to do with these issues?
2/25 13:56:30.634 bd4 Sw: DDDisk_Dispatch_WriteSameAndUnmap: The current ( non de-duplicated device ) does not support the Unmap command!
2/25 14:03:07.284 fec Sw: DDDisk_Dispatch_WriteSameAndUnmap: The current ( non de-duplicated device ) does not support the Unmap command!


It says that it's a non de-duplicated drive, but when I created it as a deduped drive.
jcrowe
 
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Re: Dedupe .spspx keeps growing

Postby anton (staff) » Wed Feb 26, 2014 9:46 am

V6 or V8 dedupe engine?

We keep all the transactions so unless "delete" is enabled the whole "history" for writes is kept.

With V8 you need to "trim" dedupe volume discarding overwrites and unused snapshots.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
anton (staff)
Site Admin
 
Posts: 3920
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands

Re: Dedupe .spspx keeps growing

Postby Alex (staff) » Wed Feb 26, 2014 2:34 pm

jcrowe,
We have added this functionality. New build with full support for unmap commands will be published today or tomorrow.
Best regards,
Alexey.
User avatar
Alex (staff)
Staff
 
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Re: Dedupe .spspx keeps growing

Postby eliripoll » Wed Feb 26, 2014 4:29 pm

HI JCrowe,

Glad your testing to, Sounds like you are seeing the same thing I observed as well, the SPspx file gradually grows even with Idle VM's until its more than the data itself. I also reported this observation as well. you can check out in URL below (which also includes another link from another reported posting)

http://www.starwindsoftware.com/forums/beta-f17/deduplication-for-problem-beta-t3352.html

Alex actually was the one working with me, so I believe they are aware and are working on getting it straightened out.

@Alex, Hi you mentioned that it may be published today or tomorrow, does that mean a new key will be provided for beta? current one ends tomorrow.
eliripoll
 
Posts: 33
Joined: Sat Dec 08, 2012 8:19 pm
Location: United states

Re: Dedupe .spspx keeps growing

Postby Alex (staff) » Wed Feb 26, 2014 4:51 pm

Yes, key will be updated too.
Best regards,
Alexey.
User avatar
Alex (staff)
Staff
 
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Re: Dedupe .spspx keeps growing

Postby Delo123 » Wed Feb 26, 2014 5:19 pm

So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
Delo123
 
Posts: 30
Joined: Wed Feb 12, 2014 5:15 pm

Re: Dedupe .spspx keeps growing

Postby anton (staff) » Wed Feb 26, 2014 8:37 pm

Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.

Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
anton (staff)
Site Admin
 
Posts: 3920
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands

Re: Dedupe .spspx keeps growing

Postby jcrowe » Wed Feb 26, 2014 8:51 pm

So would it be a good idea to run something like SDelete in windows to zero out all freed space in the OS?

anton (staff) wrote:Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.

Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
jcrowe
 
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Re: Dedupe .spspx keeps growing

Postby anton (staff) » Wed Feb 26, 2014 9:12 pm

Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)

jcrowe wrote:So would it be a good idea to run something like SDelete in windows to zero out all freed space in the OS?

anton (staff) wrote:Yes, that was a plan (c) ...

If OS passes either UNMAP or WRITE with all zeros (we do check for this particular pattern) then we'll free the block from the current active snapshot.

Delo123 wrote:So you will support Unmap in the new build?

Should somebody be looking for this:
To actually unmap / delete unused storage from ESX (5.5, 5.1 command is slighlty different) one has to run it activly in ESX Shell by something like this:
/vmfs/volumes # esxcli storage vmfs unmap -l "LUN55"

I assume this will also be needed for Starwind?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
anton (staff)
Site Admin
 
Posts: 3920
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands

Re: Dedupe .spspx keeps growing

Postby fbifido » Thu Feb 27, 2014 9:08 pm

anton (staff) wrote:Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)


Can you please post the link ?
User avatar
fbifido
 
Posts: 125
Joined: Thu Sep 05, 2013 7:33 am

Re: Dedupe .spspx keeps growing

Postby anton (staff) » Thu Feb 27, 2014 9:27 pm

The same as it was before:

http://www.starwindsoftware.com/starwind-v8-beta

Re-apply and get the key and the link.

fbifido wrote:
anton (staff) wrote:Absolutely! Or VHDX shrink sequence PowerShell-scripted for all running VMs. Doable either way. Modern OSes inside VMs are UNMAP friendly already. We'll see maybe
we'll just report we're flash storage. RC is out so we're open for ideas :)


Can you please post the link ?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
anton (staff)
Site Admin
 
Posts: 3920
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands

Re: Dedupe .spspx keeps growing

Postby jcrowe » Thu Mar 06, 2014 6:43 pm

While the RC release slowed down the growth of the .spspx files, it is adding 5-10 of these files a day. The size used now is over the size of the VM's it is supposed to be depuplicating. Is there a way to force cleanup on these?
jcrowe
 
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Re: Dedupe .spspx keeps growing

Postby Alex (staff) » Fri Mar 07, 2014 10:25 am

We are working on functionality for executing defragmentation by user command. Also automatic defragmentation scheduler settings will be revisited later.

Please let us know, what kind of tasks are executed on your VMs? Is it SQL database, Web-server or mail server, some specific software?
Best regards,
Alexey.
User avatar
Alex (staff)
Staff
 
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Re: Dedupe .spspx keeps growing

Postby jcrowe » Fri Mar 07, 2014 6:28 pm

I'm running a little bit of everything. The highest IO VM's are a MySQL and MSSQL server, following closely by a UTM firewall.

Alex (staff) wrote:We are working on functionality for executing defragmentation by user command. Also automatic defragmentation scheduler settings will be revisited later.

Please let us know, what kind of tasks are executed on your VMs? Is it SQL database, Web-server or mail server, some specific software?
jcrowe
 
Posts: 22
Joined: Wed Jan 08, 2014 1:49 am

Re: Dedupe .spspx keeps growing

Postby Alex (staff) » Tue Mar 11, 2014 10:29 am

Thank you!
Best regards,
Alexey.
User avatar
Alex (staff)
Staff
 
Posts: 177
Joined: Sat Jun 26, 2004 8:49 am

Next

Return to Beta

Who is online

Users browsing this forum: No registered users and 2 guests