Target Disk Fragmentation

[Brandon]

I'm experiencing a pretty severe issue with fragmentation during large/long file copies from an initiator.

Scenario:
Target System - Starwind 2.6.3
3.6TB RAID5 - (11) 400GB - 3ware 9500S-12
1.5TB Sparse Image

Initiator System - MS Initiator 1.06
Copying from external 250GB HDDs through the iSCSI connection to the 1.5TB volume.
The copy starts fine but gradually slows down and eventually stops all together.

The target array fragments severely as the file copy progresses. It appears to be a file system issue on the target but I would like to hear other people’s impressions.
I'm going to be trying different strip size and cluster sizes on the array.

Has anyone else experienced these types of issues?

Thanks,
Brandon

[Brandon]

I should note that some colleagues have been testing a similar target setup but have been using WinTarget (String Bean) instead of StarWind.

They have reported very little fragmentation during long data writes to an image file on the array.
I have not witnessed every detail personally but it sounds like something software related during the data writes.

Any ideas?

Brandon

[DJ_Datte]

Hello!

I will have to sign this one too!

But, I have found a way around it, dont use sparse image. With your writing performace, you should be able to make a flat image in about a hour, and be done with it.

I've had similiar problems, with fragmentation running in thousands parts of a file, making performance become like using raided floppydrives

I belive its a windows issue tho, not a starwind issue. It might be correctable by starwind tho, if for example, when starwind knows a 100mb file is incomming, it will allocate all 100mb of the sparse file, before it begins the write to disk!

/Damir

[anton (staff)]

Absolutely correct! StarWind can work with non-allocated images, other cannot. This (sparse files) leaves you with comparably free disk but you have a penalty of extra allocation write per every physical iSCSI write. Another thing - it's a good idea to have cluster size of 64KB+ on volume holding iSCSI image.

[Tim]

How do I create a multi-gigabyte zero filled image file?

Not sparse but fully allocated for maximum performance.

[Val (staff)]

How do I create a multi-gigabyte zero filled image file?

Not sparse but fully allocated for maximum performance.

Tim,

You could use our mksparse.exe with '-o' parameter.

F.e. mksparse.exe -o file.img 1024G

[Tim]

You could use our mksparse.exe with '-o' parameter.

Ah, so that's what the '-o' parameter means.
A quick look around didn't reveal any documentation for the mksparse program.

Thanks.

[Val (staff)]

You could use our mksparse.exe with '-o' parameter.

Ah, so that's what the '-o' parameter means.
A quick look around didn't reveal any documentation for the mksparse program.

Thanks.

Just start the utility without parameters and you will see:

mksparse: Image file creation utility Version 1.1

Usage: mksparse [-o|-extend] <filename> size[k|M|G]

Examples:
mksparse file.img 10M -- create a new 10MB image (sparse file)
mksparse -o file.img 10M -- create a new 10MB image (flat file)
mksparse -extend c:\temp\image.dat 100M -- extend an existing image by 100MB

Note: size will be aligned on sector boundary.

[Brandon]

DJ_Datte:

Thanks for the suggestion. The flat file fixed the problem.
I wrote 300GB of data and there was practically no fragmentation on the target. I made sure to defrag the empty flat file prior to writing any data.

Brandon,

[DJ_Datte]

Brandon:

You are welcome Dont forget that you can also defragment the actual drive on the initiator too from time to time, just like a normal drive =).

Hey, I got a question for you. When you defragmented, did you also notice that windows writes a part of the file on the start of the disk, then makes a gap thats about 20% of the capacity, then continues writing the file, so there is no way making so you dont have the flat image atleast in 2 parts (which isnt bad, just interesting ntfs behaviour).

/Damir

[Tim]

Hey, I got a question for you. When you defragmented, did you also notice that windows writes a part of the file on the start of the disk, then makes a gap thats about 20% of the capacity, then continues writing the file, so there is no way making so you dont have the flat image atleast in 2 parts (which isnt bad, just interesting ntfs behaviour).

That'll be the MFT getting in the way. It normally sits near the start of the partition.
If you are only using the drive for large image file(s) then you can recover some space from the MFT (can't remember how ATM).
You only need a 'normal' size MFT when you there are lots of files in the filesystem.

[DJ_Datte]

Uhm, a 200GB MFT ?

/Damir

[Tim]

Uhm, a 200GB MFT ?

Maybe, depends on the total partition size.


Standard MFT size is 12.5% of total free space when formatted.
There is a registry option to change this to 25%, 37.5 or 50%.

[anton (staff)]

Just a hint: use raw volumes w/o file system at all. Would save some place and could avoid "piggybacks" with growing cache size on server machine.

Uhm, a 200GB MFT ?

Maybe, depends on the total partition size.


Standard MFT size is 12.5% of total free space when formatted.
There is a registry option to change this to 25%, 37.5 or 50%.

[DJ_Datte]

I've tried raw volumes.. But do they work with the IBV plugin ? As I understand it, they dont ? Or am I wrong ?

/Damir