Some questions

[Lou]

Hi...

What's the difference between turning "asynchronous mode" on and off?

Are reads and writes to SPTI devices or image files are buffered?

Why is mksparse always creating heavily fragmented files on the host system?

[Val (staff)]

Hi...

What's the difference between turning "asynchronous mode" on and off?

Are reads and writes to SPTI devices or image files are buffered?

Why is mksparse always creating heavily fragmented files on the host system?

Hi,

1) The Asynchronous mode of the ImageFIle plug-in allows several disk requests to be executed at a moment. The Synchronous mode allows only one disk request to be current and other requests must wait until the current one finishes.
On a RAID or SCSI disk devices the async mode could give you much better performance than using the synchronous mode.

2) There is no buffering for SPTI devices. SPTI requests are executed synchronously.

3) By defaule mksparse,exe creates SPARSE files on NTFS partitions.
You need to use -o parameter to create a plain image file.

[Lou]

Hi,

1) Are there are reasons not to use asynchronous mode?
2) Are reads and/or writes to image files buffered?
3) I always used mksparse -o but the whole file is even on a blank disk heavily fragmented. Do you just open the file for writing without telling the win32 api how big the file would be?

[Val (staff)]

Hi,

1) Are there are reasons not to use asynchronous mode?
2) Are reads and/or writes to image files buffered?
3) I always used mksparse -o but the whole file is even on a blank disk heavily fragmented. Do you just open the file for writing without telling the win32 api how big the file would be?

Hi,

1) The async mode uses more system resources and in some disk configurations (like an ImageFile stored on a IDE hard drive) there is no benefit of using the mode and sometimes the resulting device's throughput is lower than for the Sync mode.

2) No. There is no buffering for ImageFile devices.

3) We'll check the issue with file fragmentation in mksparse.exe. Thank you for the report.

[Lou]

2) No. There is no buffering for ImageFile devices.

Are there are plans for supporting this in the near future? Would speed up some things very much.

[Val (staff)]

2) No. There is no buffering for ImageFile devices.

Are there are plans for supporting this in the near future? Would speed up some things very much.

Hi Lou,

The ImageFile plug-in already has such a feature, but it's inaccessible from the user unterface. - We'll add this to the next version.

To allow buffering for an existing ImageFile target, you need to edit the config file and add the following parameter to the device's rule:
-buffering:"yes"
After restarting the service the target device will use buffering for disk operations.

Please let us know whether this give any performance improvements in your tests.

Thank you!

[Lou]

Did some quick IOMeter benchmarks with different settings:

First column is IO/s, second one is MB/s. Maybe I'm making some typical workload and test the whole thing again.

Code: Select all

               sync, unbuffered     sync, buffered     async, unbuffered     async, buffered            SMB	
Linear Read     201,12  20,13 	    158,71  15,94 	  189,91   19,45 	    153,03  14,92       154,20   15,46 
Linear Write	 155,46  15,97 	     85,74   8,70 	  160,19   15,77 	     83,08   8,24 	    66,93    6,90 
Random Read	   58,04   5,85 	     56,12   5,72 	   55,41    5,61 	     52,99   5,32       119,09   11,71 
Random Write	  92,10   8,96 	     41,69   4,12 	   90,29    8,99 	     39,12   4,19 	    33,43    3,58 

[Val (staff)]

Did some quick IOMeter benchmarks with different settings:

First column is IO/s, second one is MB/s. Maybe I'm making some typical workload and test the whole thing again.

Lou,

Thank you for sharing your test results!

It seems your test ImageFile is placed on a single drive's partition (is it an IDE drive?).
So there is no benefit from using the Async mode.

If the ImageFile is stored on a RAID partition, using the Async mode should show much better results than the Sync one...

[Lou]

Hi,

Thank you for sharing your test results!

No problem. Will do some further benchmark with kind of sophisticated access patterns to see where the improvements of buffering are.

It seems your test ImageFile is placed on a single drive's partition (is it an IDE drive?).
So there is no benefit from using the Async mode.
If the ImageFile is stored on a RAID partition, using the Async mode should show much better results than the Sync one...

Imagefile is placed on a RAID5 with 4 IDE drives and a Promise SX4000 RAID controller.

I don't know anything about Starwind internals, but if you are handling all requests for a target within a single thread there is - as far as I know - a strictly sequential file IO and thus no benefit. The readings are to close together for a valid conclusion.

I wonder why the results of buffering are so worse. If I split up the test into different request sizes, the small request sizes until 32k are speed up while the larger request sizes are slowed down when using buffering.

[Val (staff)]

Hi,

It seems your test ImageFile is placed on a single drive's partition (is it an IDE drive?).
So there is no benefit from using the Async mode.
If the ImageFile is stored on a RAID partition, using the Async mode should show much better results than the Sync one...

Imagefile is placed on a RAID5 with 4 IDE drives and a Promise SX4000 RAID controller.

I don't know anything about Starwind internals, but if you are handling all requests for a target within a single thread there is - as far as I know - a strictly sequential file IO and thus no benefit. The readings are to close together for a valid conclusion.

I wonder why the results of buffering are so worse. If I split up the test into different request sizes, the small request sizes until 32k are speed up while the larger request sizes are slowed down when using buffering.

RAID-5 is not the best mode if you need good performance.
RAID-1+0 would show better throughput and redundancy.

Also I'd like to point out that the total performance depends on the RAID's strip size and sizes of your test workload packets.

Recently I've read a very good document from MS called "Disk Subsystem Performance Analysis for Windows". http://www.microsoft.com/whdc/device/st ... _perf.mspx
The white-paper gives a lot of useful details on how RAIDs work in context of Windows NT OSes.

You are right, we do use sending R/W requests to the device from a single thread. But the file is opened as overlapped and the Async mode allows multiple pending requests to the file.
NTFS should definitely have benefit from overlapped I/O to file placed on a RAID-x partition. At least it does it with SCSI disk arrays...

Buffering is only good for rather short requests much less than the RAID's strip unit size especially for sequentional I/O.
But when the request size is near or more than the strip unit size, there is no benefit in using buffering for sequentional or random I/O operations.

[Lou]

RAID-5 is not the best mode if you need good performance.
RAID-1+0 would show better throughput and redundancy.

Sure, but RAID5 delivers us enough performance for our projected applications. The main objective is storage consolidation.

Also I'd like to point out that the total performance depends on the RAID's strip size and sizes of your test workload packets.

I also agree on this but I cannot tell our applications to use a particular workload size

You are right, we do use sending R/W requests to the device from a single thread. But the file is opened as overlapped and the Async mode allows multiple pending requests to the file.
NTFS should definitely have benefit from overlapped I/O to file placed on a RAID-x partition. At least it does it with SCSI disk arrays...

We'll check this next week with different hardware. We also want to take readings from other iSCSI target software.

Buffering is only good for rather short requests much less than the RAID's strip unit size especially for sequentional I/O.
But when the request size is near or more than the strip unit size, there is no benefit in using buffering for sequentional or random I/O operations.

Our expectations on the benefits of buffering are serving clients doing heavily localized IO, mostly random. The buffering should improve these requests with write-back caching and readahead. Also, performance on other request types (large sequential or random IO) mustn't drop down below using no buffering.

[Val (staff)]

Also I'd like to point out that the total performance depends on the RAID's strip size and sizes of your test workload packets.

I also agree on this but I cannot tell our applications to use a particular workload size ;-)

Do your applications open files with local buffering?

You are right, we do use sending R/W requests to the device from a single thread. But the file is opened as overlapped and the Async mode allows multiple pending requests to the file.
NTFS should definitely have benefit from overlapped I/O to file placed on a RAID-x partition. At least it does it with SCSI disk arrays...

We'll check this next week with different hardware. We also want to take readings from other iSCSI target software.

Ok, please let us know about the results.

Buffering is only good for rather short requests much less than the RAID's strip unit size especially for sequentional I/O.
But when the request size is near or more than the strip unit size, there is no benefit in using buffering for sequentional or random I/O operations.

Our expectations on the benefits of buffering are serving clients doing heavily localized IO, mostly random. The buffering should improve these requests with write-back caching and readahead. Also, performance on other request types (large sequential or random IO) mustn't drop down below using no buffering.

Yes, I agree with you that if an application reads/writes to a list of disk blocks, it should help if the blocks were cached.
BUT this is only true for files that are opened without buffering in the client host's filesystem. If the file is buffered on the initiator's side, it's simple waste of server's resources to use double buffering.

[Lou]

As our applications are commercial available software, we don't have any influence on their buffering behaviour. But it's important that the iSCSI targets honours SCSI sync cache requests to ensure that critical data is flushed to disk.

I couldn't do any benchmarks last week cause I had to travel to brazil (fine weather ). Will do them this week (unless I don't have to travel otherwhere).

My idea of target buffering was, that if I have a target with much of unused memory, I could utilize this to cache initiators requests. Sure, most applications on initiator are using local buffering, but expensive SAN arrays do target caching, too. And on 'cheap' iSCSI target servers the buffering should give me an advantage when writing to RAID5 arrays.

I really want to know why there are such a performance decrease with activated buffering. This shouldn't happen, at least it didn't happen with our experimental linux iSCSI target. Maybe some trashing in Windows' cache algorithm. Maybe I get someone of our experts.

[Val (staff)]

Lou,

As our applications are commercial available software, we don't have any influence on their buffering behaviour. But it's important that the iSCSI targets honours SCSI sync cache requests to ensure that critical data is flushed to disk.

You could use some monitoring tool (for example, SysInternals' FileMon) to see whether your applications use buffering and what sizes of disk requests are there...

My idea of target buffering was, that if I have a target with much of unused memory, I could utilize this to cache initiators requests. Sure, most applications on initiator are using local buffering, but expensive SAN arrays do target caching, too. And on 'cheap' iSCSI target servers the buffering should give me an advantage when writing to RAID5 arrays.

Any buffering has its pros and cons. If an iSCSI target tries to buffer data that the initiator demands to be non-buffered, the target must be fail-tolerant enough to ensure the data is written to a persistent storage.

I really want to know why there are such a performance decrease with activated buffering. This shouldn't happen, at least it didn't happen with our experimental linux iSCSI target. Maybe some trashing in Windows' cache algorithm. Maybe I get someone of our experts.

I guess Linux and Windows uses different caching algorithms in their filesystems.