LSFS overhead & defragmentation

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
robnicholson
Posts: 359
Joined: Thu Apr 14, 2011 3:12 pm

Wed Sep 24, 2014 3:41 pm

Doing some tests with LSFS - bet you hadn't guessed that :D

Created new 500GB LSFS storage on two-node HA and started copying some test files onto it using robocopy via Windows 2012 server/iSCSI MPIO. 228GB of data has been copied:
sshot-58.png
sshot-58.png (28.39 KiB) Viewed 3102 times
And there are 275GB of SPSPX files on the server:
sshot-59.png
sshot-59.png (23.6 KiB) Viewed 3102 times
So the SAN is using 17% more space that the amount of data copied. NOTE: no files have been deleted/overwritten - this is a completely fresh copy.

Is this kind of overhead expected? Slightly higher than I expected but I'm aware that there are many more writes when copying files like this than just the data itself. 363,926 files have been copied which will have resulted in many of additional writes to the directory. Even when the server is idle, there are writes going onto $LogFile, $Bitmap & $Mft.

Thinking about the master file table alone, the same block will have been re-written many times as individual new directory entries are written into the same directory block. In effect, the same block will have been overwritten many times. Will defragmentation pick this up and eventually free that space or does defragmentation only work at the 5GB SPSPX level? Or does it work at a more granular level, i.e. if a block has been flagged as deleted/overwritten, are blocks from higher up moved down so that eventually an entire 5GB SPSPX file will become free and can be deleted. Is that a way to see defragmentation in action, you'll end up with gaps in the SPSPX numbering system?

So many questions ;-)

Cheers, Rob.
robnicholson
Posts: 359
Joined: Thu Apr 14, 2011 3:12 pm

Wed Sep 24, 2014 3:44 pm

Also, is defragmentation the correct phrase? Because if it moves blocks from higher up over deleted/free blocks lower down, this could actually create defragmentation in that whereby blocks were possibly sequential on disk, they are now scattered all over the deleted/free blocks.

Cheers, Rob.
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Mon Sep 29, 2014 8:59 pm

Rob, that is a great job, however we already have few significant improvements under development that should fix this and few other aspects. Stay tuned :)

BTW, have you singed in to our Beta-testing program? Let me know if not and I will do that for you personally :)
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
robnicholson
Posts: 359
Joined: Thu Apr 14, 2011 3:12 pm

Tue Oct 07, 2014 11:20 am

BTW, have you singed in to our Beta-testing program?
Well you sent me the t-shirt so I assume so?? ;-) But I haven't personally signed up for anything.

Cheers, Rob.
User avatar
Anatoly (staff)
Staff
Posts: 1675
Joined: Tue Mar 01, 2011 8:28 am
Contact:

Mon Oct 13, 2014 10:02 am

Yup)
Best regards,
Anatoly Vilchinsky
Global Engineering and Support Manager
www.starwind.com
av@starwind.com
Post Reply