Smaller Jumbo Frames?

Public beta (bugs, reports, suggestions, features and requests)

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
User avatar
Aitor_Ibarra
Posts: 163
Joined: Wed Nov 05, 2008 1:22 pm
Location: London

Fri Sep 17, 2010 1:48 pm

In my brief performance tests of latest beta, I don't see any difference between 4088 byte and 9016 byte jumbo frames. With either, I was able to get to 550MB/sec reads from cache over 10GbE, and one test when I had two clients testing the same target simultaneously got to 700+ MB/sec.

So, if your clients are using the default windows allocation unit of 4096 bytes (which is more than a 4088 byte frame!) and you use 9k frames, and on the starwind side your disk is also formatted as 4096 byte allocation units, is the ethernet 9016 byte packet padded with zeroes, or will it have two allocation units worth of data in it? Or is the allocation unit size irrelevant because iSCSI groups IOs into larger chunks, or whatever?

Basically I'm looking for a reason for there not being much of a difference between 4088 byte and 9016 byte frames, and wondering what the optimal setting is really going to be given that most IOs are going to be for multiples of 4096 bytes. In the end I actually care more about IOPs than transfer rates, but it's that 4096 byte windows default that I want to optimise for.
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Sat Sep 18, 2010 5:27 pm

Jumbo frame size is entirely implementation-specific thing and depends on the selected configuration greatly. The same hardware but different drives can change everything (Intel Pro 1GbE hardware for example, 16KB Vs. 9KB performance chances both sides from release to release). Also intermediate network hardware (switches) have great influence. So proper answer is: TEST BEFORE YOU PUT INTO PRODUCTION :)
Aitor_Ibarra wrote:In my brief performance tests of latest beta, I don't see any difference between 4088 byte and 9016 byte jumbo frames. With either, I was able to get to 550MB/sec reads from cache over 10GbE, and one test when I had two clients testing the same target simultaneously got to 700+ MB/sec.

So, if your clients are using the default windows allocation unit of 4096 bytes (which is more than a 4088 byte frame!) and you use 9k frames, and on the starwind side your disk is also formatted as 4096 byte allocation units, is the ethernet 9016 byte packet padded with zeroes, or will it have two allocation units worth of data in it? Or is the allocation unit size irrelevant because iSCSI groups IOs into larger chunks, or whatever?

Basically I'm looking for a reason for there not being much of a difference between 4088 byte and 9016 byte frames, and wondering what the optimal setting is really going to be given that most IOs are going to be for multiples of 4096 bytes. In the end I actually care more about IOPs than transfer rates, but it's that 4096 byte windows default that I want to optimise for.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
deiruch
Posts: 35
Joined: Wed May 25, 2011 12:16 pm

Wed May 25, 2011 1:15 pm

Awakening zombie thread!

Overhead of a 9k frame compared to a 4k frame is only a little lower. And if you enable jumbo frames not all frames are padded to a certain size - the max size is just bigger. So 20 byte packets won't suddendly be 9k after enabling jumbo frames. It's just that 9k packets don't have to be fragmented & reassembled from multiple 1.5k parts.

Having 4k jumbo frames means 2 packets per 4k read (Because of iScsi overhead and data), if you enable 9k frames a single packet should be enough. Grouping cannot be done with iSCSI.
User avatar
anton (staff)
Site Admin
Posts: 4008
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Wed May 25, 2011 4:07 pm

Sure. MTU is *Maximum* Transfer Unit and not constant.

Again... The answer is "it depends". So both 4KB and 9KB packets should be checked before putting whole configuration into production.
deiruch wrote:Awakening zombie thread!

Overhead of a 9k frame compared to a 4k frame is only a little lower. And if you enable jumbo frames not all frames are padded to a certain size - the max size is just bigger. So 20 byte packets won't suddendly be 9k after enabling jumbo frames. It's just that 9k packets don't have to be fragmented & reassembled from multiple 1.5k parts.

Having 4k jumbo frames means 2 packets per 4k read (Because of iScsi overhead and data), if you enable 9k frames a single packet should be enough. Grouping cannot be done with iSCSI.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
Post Reply