Grossly bloated deduplicated target

Public beta (bugs, reports, suggestions, features and requests)

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Mon Aug 06, 2012 3:04 pm

This morning I came in and found that my monitoring system was giving me alarms on my SAN server getting close to running out of diskspace.

I have Starwind 5.8 installed on Windows 2008 R2. My target is on a 1.22TB raid array by itself (there is nothing else on that volume). I've got about 62.5GB free on that volume, even though my target is only 1TB.

I look in my vmware server (ESXI 5), which for right now is the only one serving VMs off of that target, and my VM Storage is show 1TB of capacity and about 838GB free!

What's using up all this disk space and how do I correct it? :(

See my attached screenshots:
Storage volume disk usage
Storage volume disk usage
SWSS1.jpg (53.64 KiB) Viewed 15514 times
Data files being used by Starwind
Data files being used by Starwind
SWSS2.jpg (87.61 KiB) Viewed 15517 times
Starwind target configuration
Starwind target configuration
SWSS3.jpg (98.01 KiB) Viewed 15520 times
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Mon Aug 06, 2012 3:05 pm

One more screenshot (I could only attach 3 in my last post)
VMware storage use
VMware storage use
SWSS4.jpg (34.26 KiB) Viewed 15512 times
:shock:

Please help.
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Mon Aug 06, 2012 6:23 pm

It looks like my growth is mostly going into the .spbitmap and .spdata files, with a very small amount of grown in the .spmetadata file. Since my original post, I'm down another 1.2GB, with about half of that space going into the .spbitmap and .spdata files.

It also looks like the gradual consumption has been going on for quite some time. The VMs that are on that machine don't really change much in terms of disk usage, so I'm not really sure what's going on. There was a big drop late in June and I think that's when I migrated some VMs off of another host onto this storage, but I migrated them back after a couple of days, so they are no longer in the SAN.
Graph showing free space on target volume
Graph showing free space on target volume
SWSS5.jpg (45.55 KiB) Viewed 15486 times
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Tue Aug 07, 2012 12:43 pm

Just one question before we proceed - you have written the data,anad deleted it,a after writen some other data and deleted it again, correct?
I`m asking because for now we do not have block re-use or block deletion support, which means that if data will be erased, it doesnt mean that the block, where it was stored will disappear, moreover, nothing will be able to be stored there (on them). This will definitely be fixed in next versions. Stay tuned.

So, if my assumption is correct then the only way that I see now is to simply create new drive (device) and move all the data on it.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Tue Aug 07, 2012 2:21 pm

Thanks for the reply, Anatoly.

Wow, if I'm understanding you correctly, that's kind of a big bug. :| Does this only apply to deduplicated disks? I don't remember this being an issue on other installs I have done where I have used some other type of starwind disk.

Going by my graph, it only gradually uses up space. Except for the big dip late in June, where I moved some VMs around and they were temporary stored on this SAN. So in that case, what you're saying makes sense because after they got moved back off of that SAN, the space was never returned.

As for the gradual decrease...the VMs that are stored there are mostly static VMs. There is a Windows terminal server on there, which only gets very light use, but I suppose as users log in their profile gets copied to the server and then deleted when they log off, so that would explain some usage.

There is also a VM that serves as a Windows DFS recplication point on there which does replicate from some active data. Depending on how that works, I suppose the replica is written and then the data is changed on another server and the new data is written and the old replica is deleted. Maybe that explains the gradual free space decline.

Sounds like I've got alot of work to do pretty soon. :(
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Wed Aug 08, 2012 7:58 pm

I'm also finding that copying the data off and creating a new target is more difficult than it sounds in an free ESXi environment. Apparently transferring files (the actual vmware files themselves) off of ESXi using SCP or Veeam's FastSCP is very slow (10-20MB/s), I guess by VMware's design from what I'm finding in google searches, so I'm in a bit of a spot now.

Not only that, but when they copy to my workstation, if they are thin provisioned disks in VMware, they seem to convert to thick disks during that process. I'm not that far yet, but I'm hoping that when I go to put them back into VMware and onto storage, I will have an option to upload them back as thin provisioned disks again...but with how my luck is going so far on this, I doubt it. :(

Do you guys have any recommendations on how I can speed up that process and/or to keep my thin provisioning I have set up currently in my vmware disks?
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Fri Aug 10, 2012 10:33 am

You absolutely should NOT use thin-provisioned disks with deduplication. Basically you make double work for everyone. Deduplication IS thin-provisioning by design. Layering extra layer of TP over it makes data more scattered, more random I/O => low performance.
ralphw wrote:I'm also finding that copying the data off and creating a new target is more difficult than it sounds in an free ESXi environment. Apparently transferring files (the actual vmware files themselves) off of ESXi using SCP or Veeam's FastSCP is very slow (10-20MB/s), I guess by VMware's design from what I'm finding in google searches, so I'm in a bit of a spot now.

Not only that, but when they copy to my workstation, if they are thin provisioned disks in VMware, they seem to convert to thick disks during that process. I'm not that far yet, but I'm hoping that when I go to put them back into VMware and onto storage, I will have an option to upload them back as thin provisioned disks again...but with how my luck is going so far on this, I doubt it. :(

Do you guys have any recommendations on how I can speed up that process and/or to keep my thin provisioning I have set up currently in my vmware disks?
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Fri Aug 10, 2012 1:14 pm

anton (staff) wrote:You absolutely should NOT use thin-provisioned disks with deduplication. Basically you make double work for everyone. Deduplication IS thin-provisioning by design. Layering extra layer of TP over it makes data more scattered, more random I/O => low performance.
That's good to know now. :)
Anatoly (staff) wrote:I`m asking because for now we do not have block re-use or block deletion support, which means that if data will be erased, it doesnt mean that the block, where it was stored will disappear, moreover, nothing will be able to be stored there (on them). This will definitely be fixed in next versions. Stay tuned.
Does this only apply to deduplacted disks? I need to know this so that I choose the right disk when I recreate the target.
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Fri Aug 10, 2012 3:17 pm

Yes, you can find this when running DeDupe ddevice only.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
ralphw
Posts: 17
Joined: Mon Mar 26, 2012 7:22 pm

Fri Aug 10, 2012 3:36 pm

Okay, thanks. I'll create some other disk type and maybe give that a try again when that issue gets worked out. But for now, at least I have a plan.
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Mon Aug 13, 2012 8:21 am

Always glad to help you!

BTW, if this is not production environment you are welcomed to install version 6 that have block re-use.
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
dtrounce
Posts: 10
Joined: Wed Sep 05, 2012 5:34 pm

Thu Sep 06, 2012 4:08 pm

1. I'm using V6.0.4768. The "deletion support" / block re-use option is marked "This feature is experimental".

I'm assuming that this means something like "We've tested it pretty extensively and not found problems. However we don't want to be held responsible if you find an bug and it wipes all your data". How comfortable would you be using deletion support in production, now or soon?


2. I notice that using this version with deduplicated HA targets, the .spdata files seem to constantly grow (by several GBs) even >18 hours after I finished writing the last data. The files are mostly Windows Server 2012 VHDX files for VMs, which are test installs so are pretty empty and are mostly idle. I suppose there may be Windows background processes doing a little work - they have test guest clusters inside them - but they too should be pretty stable.

I was initially assuming that this is due to data from one target still being copied to it's partner. But >18 hours in, that seems unlikely - the targets should have synced by now (I'd be worried if they hadn't, for HA). I have fast disk arrays with a GbE connection between the hosts. The starwindservice.exe on both hosts is frequently reading ~500KB/sec and writing ~50KB/sec to the .spdata files and sending network traffic to its partner. What is going on? When should I expect my .spdata files to stop growing?
User avatar
Max (staff)
Staff
Posts: 533
Joined: Tue Apr 20, 2010 9:03 am

Mon Sep 10, 2012 8:59 am

Hi David,
The block re-use feature is still experimental so there are several things to be fixed before releasing it.
We're currently working on this and expect an updated version to be available by the end of the month.
Max Kolomyeytsev
StarWind Software
dtrounce
Posts: 10
Joined: Wed Sep 05, 2012 5:34 pm

Tue Oct 09, 2012 4:07 pm

I have a 100GB de-deduplicated HA target using block reuse. It has 85GB of heavily duplicated VM data which started at 30GB deduplicated.

But after a few weeks of idle activity, this has grown to a pair of 208GB .spdata files, and shows no sign of stopping growing!

I am using 6.0.4768. I see the latest download is now 6.0.4837. Does this fix this problem?
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Tue Oct 09, 2012 5:55 pm

Did you enable delete (makes sense for VM production, no-delete is for "collection" or data backups).
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
Post Reply