nvme initiator tuning

[Xavier]

on iscsi initiator it's possible to modify

“MaxTransferLength”=dword:00040000
“MaxBurstLength”=dword:00040000
“FirstBurstLength”=dword:00040000
“MaxRecvDataSegmentLength”=dword:00040000

is there a way to do same for nvme initiator ?

[yaroslav (staff)]

Hi,

Check under
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\StarNVMeoF\Parameters

[Xavier]

Hello

it contains

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\StarNVMeoF\Parameters]
"Auto"=dword:00000000
"BusType"=dword:0000000a
"IoLatencyCap"=dword:00000000
"IoFlags"=dword:00000003

so i just need to add others keys at same level ?

[yaroslav (staff)]

Greetings,

No, you need only to set up MaxTransferLength to the optimal value (e.g., 8K or 4K).

[Xavier]

i'm evaluating nvme initiator against iscsi.

On my case i have esxi using mellanox with sr iov , vm use VF to have direct mellanox access

configuration using iscsi use large block as my application did large io (lets say near 128k on read , 256k on write)

i have best result on iscsi using transfertlength @256k,

i just try 4k with starwind and results are not good , what is the max size i can use ? , by default it's seems to be 64k (it's what i see on storage frontend)

[yaroslav (staff)]

Thanks for your update. Feel free to play around with the parameters as all of them need careful alignment for each system.
8K block size was noticed to be the best fit for certain targets. You can try using any other value for tests.

[Xavier]

ok thx

i already tested 4k 8k and 16k , i see large improvement against default value with 16k so you're right i need to dig on this

[yaroslav (staff)]

Greetings,

Please keep me posted. Thanks!

[Xavier]

feedback on my side

iscsi vs nvme on my particular case i see 4% increase on bandwidth for nvme with 16k (the one that give me better result) against iscsi configure to 256k
increasing to more than 16k with nvme initiator is counterproductive from what i see
but also an increase of 10% of cpu usage
Not enough for me to switch to nvme for my particular use case

Thanks for your support

[yaroslav (staff)]

Thanks for your update. You are always welcome.
Were you using NVMe-of over TCP or RDMA? Also, could you please tell me more about the network topology of the system and the underlying storage?

[Xavier]

We use nvme over roce ,
storage is huawei 5510 configured with hybrid pool (ssd and nlsas of 14TB),
between storage and esxi we use huawei ethernet 25Gb switchs also (two TOR),
we have mellanox connectx4 as smart nic on esxi,
we use sr iov to use vf on vm and two nic configured on vm side each with particular vlan.
Storage is thin , size 48TB, one lun only , io profil is 256k io size for write and variable on read with majority of iops between 8k and 16k, full random for write, sequential for read
we did this test with nvme to see which improvment we can see with cost associated to add smart io nic card also on storage array

[yaroslav (staff)]

Thanks for your update.

storage is huawei 5510 configured with hybrid pool (ssd and nlsas of 14TB),

Did you test underlying storage performance separately?

Storage is thin

Based on my previous experience, thin storage eats a lot of performance due to additional i/o for provisioning storage.

What I am saying is that the storage settings can be optimized too.

[Xavier]

Hello

We compare same storage configuration only changing protocol iscsi vs nvme over roce

This storage only support thin devices, and i agree with you it could affect performance during unmap command, but on our side we have desactivate it to not put more pressure on backend

[yaroslav (staff)]

Xavier,

Quite an unfortunate limitation.