Offloaded Data Transfers (ODX) bug appends files with NULs?

[vladimir@ngsoft.com]

Good day,

I've encountered very interesting issue:
We have 3-nodes StarWind vSAN, which provides virtual disks by iSCSI to multiple pairs of Windows 2012 R2 servers - each pair works as a Failover Cluster.
Each Failover Cluster has a fileserver role, which exposes directories on a virtual disk as shared folders, using SMB Transparent Failover.

Now, if I put a file of ~10MB on such shared folder, and then copy it via SMB - to the same shared folder or another one - then the file gets appended with zeroes. Its size grows up to the integer number of virtual disks sectors * sector's size.

Example:
* We have a file of 10025196 bytes. Its “size on disk” is 10027008 bytes = 2448 * 4096, because the iSCSI-mounted disk is formatted with NTFS, using clusters of 4096 bytes. But the SectorSize parameter for the disk itself is 512.
* After copy from FC share to FC share it grows to 10025472 bytes – growth of 276 bytes, which is less than 512. Note that the new size is actually an integer and odd number of 512-byte chunks: 10025472 = 19581 * 512
So the file is appended with NULs up to the nearest 512-bytes-sector border, but no more.
* If I do the same on another virtual disk, with SectorSize = 4096, then the file will be appended with zeroes up to its "size on disk" - because NTFS cluster size is also 4096.

This happens only if I copy the file via SMB from the shared folder belonging to Failover Cluster's fileserver role to the same folder - or to another one on the same FC.
If I copy the file from or to another machine - it doesn't happen.
If I copy the file directly on NTFS level, bypassing SMB - it doesn't happen.
If I remove the virtual disk from Failover Cluster's storage and expose it as regular Windows shared folder - it doesn't happen.

So sounds like totally Windows bug, right? But here's the catch: if I use iSCSI disk mounted from Windows iSCSI Target server, not from StarWind - it doesn't happen.

I started checking what is so special about share-to-share file copy and found out about Offloaded Data Transfers (ODX) technology, which allows copying SAN files without even reading them on the fileserver. Then I just turned ODX off on my Failover Cluster members, as described here:
https://blogs.technet.microsoft.com/ask ... x-working/
- and voila! the bug is gone! no zeroes appended!

So it seems that there's something in the ODX interaction between FC fileserver role and StarWind. I wonder, is it a known issue?
Are there any solutions except sacrificing performance by turning ODX off?

Thanks in advance,
Vladimir.

Update:
Upgraded StarWind vSAN to the latest version - v8_build_11818. It has a separate setting in the StarWind.cfg file:
<OdxEnabled value="no"/>
While it's set to "no" - no zeroes are appended on copy. If I change it to "yes" - the problem resumes. If I change it back to "no" - disappears.
So there's definitely a bug in ODX implementation.

[Oleg(staff)]

Hello Vladimir,
Thank you for so detailed investigation.
I already escalated this case to R&D Department.