LSFS over-provisioning

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Mon Dec 02, 2019 12:01 am

Hi,

I have been using Starwind vSAN free v8.0.13182 for the last few months. I am very impressed by the speed of drive, and once I learned the correct powershell commands, administering it was fine as well.

I have used the advice here and setup a ReFS mirrored array, then a deduped LSFS device on top of that. I have adequate RAM, and I have created my LSFS device based on 200% over-provisioning.

Attached is a screenshot of the LSFS device showing just over 1TB of user data, that it is taking up 7.5TB of physical disk space. Is this expected behaviour? Is there some sort of manual garbage collection that I am meant to be running?
LSFS Device
LSFS Device
Annotation 2019-12-02 105229.png (26.96 KiB) Viewed 8187 times
How do I keep this disk usage under control?

Regards,
Daniel
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Mon Dec 02, 2019 6:01 am

As troubleshooting, I flushed the cache and then restarted the Starwind service (in a hope that it would force try defragging that device) and after 6 hours the device is now remounted, but with the same size and amount of fragmentation (820%).

Looking at the end of the server log is this:

Code: Select all

12/2 16:30:54.784133 22b0 error: Sp: --- CStarPackCoreNew::CStarPackFileAsync::SetFileSize(541065216) (line #1287) 112
12/2 16:30:54.784357 22b0 error: Sp: --- CStarPackCoreNew::CStarPackSpaceSegmentFile::OpenSuperPageFile([S:\vSAN_LSFS1\\lsfs1_15213.spspx], 4194304, 128, 0, _, 0, 0x000000D4F13FF470) (line #1699) 112
12/2 16:30:54.784381 22b0 error: Sp: *** CStarPackCoreNew::CStarPackSpaceSegmentFile::PrepareSegment (252) Failed to open file 'S:\vSAN_LSFS1\\lsfs1_15213.spspx' for segment '15213'! Status (112)
12/2 16:30:54.784399 22b0 error: Sp: --- CStarPackCoreNew::CStarPackSpaceSegmentFile::PrepareSegment(Segment index - 15213, Size in pages - 128, Page size - 4194304, 0x000001E0F25B94B0) (line #254) 112
12/2 16:30:54.784416 22b0 error: Sp: *** CStarPackCoreNew::CStarPackSpaceSegmented::PrepareSegment (2381) Failed to prepare segment '15213'! Status (112)
12/2 16:30:54.784432 22b0 error: Sp: *** CStarPackCoreNew::CStarPackSpaceSegmented::AccessSegmentOptimizerWorker (4193) Failed to prepare space segment '15213'! Status (112)
12/2 16:30:54.784443 22b0 General: MonitorStarPackSpace: Out of free space on storage. LSFS disk switched to the 'No space' IO mode. Device is read-only now.
12/2 16:30:55.383362 27fc General: *** DDDisk_Dispatch_WriteAll: Write  offset 0x3576B8 is rejected due to out of space!
12/2 16:30:55.383513 27fc General: *** DDDisk_Dispatch_WriteAll: Write  offset 0x357694 is rejected due to out of space!
12/2 16:30:55.383762 27fc General: *** DDDisk_Dispatch_WriteAll: Write  offset 0x357798 is rejected due to out of space!
Does that mean that because the underlying storage is already full, that it can't perform a defragmentation?
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Mon Dec 02, 2019 11:48 pm

Hi,

Any thoughts on what to do? At the moment the device is in read-only and I cannot use it as is. I would like to continue my evaluation, but this is a bit of a show-stopper.

Thanks
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Tue Dec 03, 2019 1:00 pm

Do you have any snapshots for that device?
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Tue Dec 03, 2019 1:45 pm

Hi,
No. I checked using the

Code: Select all

# 
# get snapshots list 
# 
if ( $device.lsSnapshotsSupported ) 
{
  foreach($snapshot in $device.Snapshots) 
  {
    Write-Host "Snapshot Id : $($snapshot.Id) Name : $($snapshot.name) Date : $($snapshot.$Date)"
  }
} 
portion of the "removesnapshot.ps1" script and nothing was enumerated.
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Wed Dec 04, 2019 2:05 pm

In the view of that, you will either need to increase the capacity of the current disk, if possible. If not, perform data migration to another location and then recreate the disk. Yet keep in mind an imagefile based one could be better for your environment. If you prefer using LSFS, you can create a disk of a smaller size, so that it meets the double storage overprovisioning requirement, which now is not completely met with you 2.49TB device.
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Thu Dec 05, 2019 6:01 am

Hi Boris,

My underlying storage is 7.48 TB in capacity, and the LSFS device is one-third of that at 2.49 TB, so I have left 200% additional capacity on the underlying storage.

The concern is that at the moment the LSFS device only contains 1.01 TB of data, yet is consuming 7.39 TB of storage; by those numbers I would have to be over-provisioned by more than 600%! From what I can see there has been none of the automatic defragmentation that is mentioned in the https://knowledgebase.starwindsoftware. ... scription/ page:
How Defragmentation works:
Defragmentation works continuously in the background. Each file segment will be defragmented when data capacity exceeds the allowed value. Maximum allowed junk rate before defragmentation process is initiated equals 60%. This value can be changed using the context menu of the device.

Data from the old fragmented file segment will be moved to another empty file segment and the old file will be deleted.

If the available disk space on the physical storage is low, the LSFS uses more aggressive defragmentation policy and slows down access speed for the end user. If there is no space on the physical drive, LSFS enters the “read-only” mode.
My user data has stayed pretty consistent over the last 2 months (hovering around 1 TB), but the "Allocated Physical Space" has been growing until it filled the underlying storage and the "Device Framentated, %" has stayed very high (currently at 820%).

I can copy the data off the read-only LSFS device and recreate the LSFS device, but if it keeps growing (the LSFS device, not the user data contained within), then in another two months I will be in the same predicament, and recreating the the LSFS device every two months isn't a sustainable practice.

Regards,
Daniel
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Thu Dec 05, 2019 1:07 pm

Daniel,

The thing about LSFS is that it does grow over time as data operations are processed. Your device's size is not just 1TB, it's 2.49TB. And taking into account the 2x overprovisioning ratio for an LSFS device you would need at least 7.47TB of the physical storage. Your value of 7.48TB is pretty close to that, so you may want to consider recreating the device with a smaller size. Otherwise, you may want to use flat image files for the whole capacity of your storage.
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Thu Dec 12, 2019 4:55 am

Hi Boris,

I have gone through the process of migrating my LSFS device to a much larger storage array (15 TB). After remounting the LSFS device I am still seeing very large amount of fragmentation (820%) - I haven't received an answer yet whether this is the expected behaviour?

I am excited by the performance of the LSFS device, and am willing to sacrifice a ratio of 1:3 data to storage space to utilise it, but I am seeing continual growth beyond 3x the size of LSFS device.
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Thu Dec 12, 2019 11:24 am

I would like to suggest you submitting a ticket with reference to this thread. I will pick it up to work on this in a more close manner. We will update the community upon resolving the case.
DWHITTRED
Posts: 17
Joined: Sun Dec 01, 2019 11:38 pm

Thu Dec 12, 2019 11:32 pm

Hi Boris,

I installed the trial in September - my 30 days has expired so there is no included support. I wanted to continue running beyond the 30 days to properly evaluate the device, and so used the vSAN free licence. My understanding is that there is no support beyond this forum? Is there a way for me to purchase once-off support for this issue?
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Mon Dec 16, 2019 8:51 am

To the best of my knowledge, the one-time assistance option is not available, yet you could always negotiate this with one of our account managers (usually the one you got the license from).
Boris (staff)
Staff
Posts: 805
Joined: Fri Jul 28, 2017 8:18 am

Mon Jan 20, 2020 8:29 am

An update for the community here.

The issue with too much storage consumption was preconditioned by disabled automatic defragmentation option at the time of the device creation.
If anybody happens to stumble upon that here is an instruction on how to overcome this:
1. Stop the StarWind service to enable device header editing. Unfortunately, it cannot be edited on the fly.
2. Locate the *.swdsk file in the device folder and make a backup of the file prior to any further operations. Open the original one with some text editor (Notepad++ is ideal for this purpose).
3. Find the line:

Code: Select all

<AutoDefrag>no</AutoDefrag>
and change its value to "yes"

Code: Select all

<AutoDefrag>yes</AutoDefrag>
4. Save the file.
5. Start the StarWind service. After the service is started, open the above header file again and check the value of that parameter (should stay "yes" if everything is done properly).

As reported by the user, storage usage went down from 10TB to 1.1TB, and fragmentation ration went down from 1100% to 26%.
Post Reply