LSFS volume grew, filled hdd, now datastore EMPTY??

Wed May 06, 2015 4:19 pm

We sincerely apologize for what happened, R&D are pro-actively working to avoid similar issues in future

dann · Tue May 19, 2015 12:18 pm

Hi, what is the status on this?

I had a crash because of that reason back in Febuary and ended up losing all my data.
Now I am in the same situation as you guys AGAIN - just for me this time it was a Power Outage and a USV failure which caused it.

My VMware Datastores are shown as EMPTY disks in VMware and my NTFS Formatted iSCSI Disk is recognized as RAW. No Data is accessible at all.

Has anyone had any luck restoring their data? How about the recovery tool - is there something I can test? I am really desperate again

Thank you very much in advance!

Kind Regards
Daniel

kkuszek · Tue May 19, 2015 12:37 pm

Reading between the lines of the comments I am to assume that since the newer versions have "theoretically" resolved this bug so you are more than welcome to better test your backups next time, pinkie swear you won't lose everything in production again.

I was never given access to any utility and given the failure mode I doubt the success of one.

I fixed my issues by buying a Synology.

Since you are so late to join the party, I wonder how many people like you had this happen but just not publicly comment or look further into their lost data?

sloopy · Wed May 20, 2015 4:06 pm

kkuszek wrote:
Since you are so late to join the party, I wonder how many people like you had this happen but just not publicly comment or look further into their lost data?

We had the same thing happen with a LUN we tested with. It probably would have never happened with production data of ours anyway because it only took 1 day to outgrow the LUN and crash it. We have just decided to steer clear of LSFS, in theory it sounds good and has had a lot of hype put out by the Starwind team, but in real world scenarios as described here people are losing data and they are constantly fixing bugs that made it past QA. I have a feeling you are right that there are many not reporting it-- their support response times are not great due to what I feel they are addressing many of the problems with LSFS for those not active on the forums.

Wed May 20, 2015 5:01 pm

Guys, we are pro-actively working on lsfs improvements. Currently overgrowing issue has been resolved, though there still is an overhead. Current lsfs specs are:
http://knowledgebase.starwindsoftware.c ... -lsfs-faq/
http://knowledgebase.starwindsoftware.c ... scription/
Again, we fully understand the impact on production data, since the last build release, there were no issues reported.

dann · Sat May 23, 2015 8:20 pm

Hi,

thanks for the response, and I'm happy you were able to resolve the overgrowing bug. But what would be way more important for me right now is a way to recover my lost data.
Of course I was able to restore most of my data from backups, but there is still a lot of stuff missing that I would like to get back, if in any way possible. Are you still working on the Recovery tool? Or did you find any way to recover data from the broken LSFS volumes?

And another thing: My LSFS volumes got corrupted by a power outage, not by overgrowing. Is this the same issue as overgrowing or will the next unexpected shutdown corrupt all the LSFS volumes, too? As far as I understood your papers the data written to disk should always be in a consistent state independend from the state of L1 or L2 Cache.

Thanks!

Daniel

kkuszek · Sun May 24, 2015 3:55 am

Dann You might be experiencing a slightly different issue, although I have not seen a log playback tool published anywhere.

Ironwolf · Thu May 28, 2015 7:56 pm

I recently went through a system failure that... well... caused the unspeakable to happen... the failure was not related to STARWIND fyi and hardware failure has been resolved, though the raid disk volumes were damaged, and files mangle.

I see some of you are trying to recover data. Have you tried to recover the data VIA Foremost? [ ... ]
It will actually ignore the container files and reads via a bit-stream. Assuming you did not use disk encryption or Data DeDuplication, it is fairly successful.

It does produce duplicate files though, but their are plenty of free utility's out their that will find and delete duplicate files.

I can tell you we ended up with about 10million files, on average it was about 1 file per 200 duplicates, and about 80% were valid files

Fri May 29, 2015 9:50 am

Hi Guys,

This tool you mentioned will be integrated into StarWind management console, it will be possible to enter maintenance mode, launch snapshot manager and check available restore points. And then mount the necessary snapshot. Currently this functionality is being developed, so we expect this in our future builds. Meanwhile your patience is highly appreciated. It is strongly recommended to upgrade to the latest build to avoid lsfs growth issues.

dann · Tue Jun 02, 2015 9:43 am

Hi,

that sounds like a solid plan! When can we expect this tool to be released?

Thanks!
Daniel

Jhon_Smith · Fri Jun 05, 2015 12:31 pm

I have heard that this option will be released in a month.

Mon Jun 08, 2015 2:34 pm

Yes, this option will be integrated in SW console. ETA - 1 month. It is possible that this feature will require additional testing, so production version may be released a bit later (about 2 weeks), but the one for testing should be ready in one month.

Francesco · Sun Jun 21, 2015 6:09 pm

I feel pretty lucky I did not lose much data due to this LSFS flop. I discovered this problem in July or August 2014 and saw no fixes. Kept a close eye on data stores and changed some to thick provisioned image files to prevent filling up the 8TB drive and losing data. Eventually, I decided to take everything off the SAN because it was just too risky.

I, also, lost data after a power outage and the UPS failed. The LSFS volumes took hours to come back online (happens even after normal shutdown) and then had to recover Exchange data; losing a few hours of email. FYI, this was the only time I ever lost data to an improper shutdown. I haven't used the SAN in a few months but really need to bring it back.

I am very wary about using StarWind and am looking for other products. I wish I could use v6 because it actually works for many people I know but will not work properly on my newer hardware with 10Gb NIC’s.

Has anyone tried Windows Storage Server 2012 R2? Or anything else?

Francesco · Sun Jun 21, 2015 6:41 pm

Oh, yes. Now I remember. The last string that made me move everything off the StarWind SAN was when memory usage was 100% because StarWind was using all 32GB of it. Data access became very slow.

Jhon_Smith · Mon Jun 22, 2015 12:33 pm

Speaking about V6, you are free to use functionality of V6 in V8.
V6 did not have LSFS and L2 cache, thus simply using thick .img files with L1 cache will provide you with V6.

Guys who lost data using LSFS, i am really sorry for you, but next time i would suggest you reading LSFS specs and configuring your system according to StarWind best practices, as well as checking for new StarWind releases. If you have done all that, you would not loose a single byte of data.

As far as it was in my case, yes, they had a moment when LSFS was growing without any control. And after that was noticed, StarWind released a fix for that in 1 week. Which was fast enough in my case.