Windows Corruption in VM

[merryweathertech]

Hi - I setup a Physical disk in SW - 60 GB, 1 GB Cache, Write-Back mode, 512 Sector, and added a replica. I mounted this disk as storage in ESXi as a datastore and then setup a Windows 2008 R2 Guest vm on this iSCSi disk. While the windows VM was on and applying updates, I started performing some failure simulations where I rebooted the secondary node, disabled the network iscsi and sync adapters on the secondary node, etc. Everytime the secondary node came back, everything started syncing and seemed to look good. Windows was still running and usable. Then I rebooted the windows VM and it would not boot - went right to blue screen error and booted into repair mode. I ran a chkdsk on the windows volume and it was full of corrupted files. Can you give me an idea of what could have happened here? Should I use Write Through mode for VMs? Is there something else that could have happened? I am doing it all over to see if I can reproduce. Thanks - Gordon

[merryweathertech]

Hi - I setup a Write Through version of the above (first post) and applied the ESXi changes to the IOPS setting and MaxDiskSize. I did a fresh VM, Windows install, and then started playing with the network and nodes while it was applying updates. Once it finished I rebooted again, and again the same issue - thousands of corrupt files and eventually won't boot after it attempts to fix them. Is it safe to say that a VM on an iSCSI instance is just too fragile? I have a SQL server database (not VM, just data files) on a SW iSCSi drive and have performed the same tests and it always continues to work, no errors, no recovery. What could be happening? Thanks - Gordon

[Oleg(staff)]

Hi Gordon,
Could you please give more details.
What networks are you using for heartbeat traffic?
Did you disable all the networks for synchronization/heartbeat simultaneously between nodes?

[merryweathertech]

Hi - I am using:

- 10 GBe QLogic dedicated to iSCSi/Heartbeat ( 10.0.10.1 and 10.0.10.2 ) - Jumbo frames enabled (9000 Frame Size to match ESXi 9000 MTU)
- 10 GBe QLogic dedicated to Sync ( 10.0.20.1 and 10.0.20.2 ) - Jumbo frames enabled (9014 as recommended by MS)

Node 2 is the .2 for both interfaces. I was performing a few things. I would disable the iSCSi port, then disable the Sync port on Node 2 only. Then wait a while, about 5 minutes, and the enable. Then once I saw that everything was synced 100%/Healthy, I would do it again in reverse order - Sync, then iSCSi. I repeated this a few times. Then after everything showed 100%/Healthy I rebooted Node 1. Once that came up, everything was 100%/Healthy and then I continued to let windows update run. After it completed, I reboot and the corruption is displayed - once it just wouldn't boot. The second time it performed a chkdsk on its own and found thousands of corrupt files. Eventually it said it fixed everything and then on a reboot it failed to boot with a bluescreen. I found a blog article that recommends disabling the 'Windows Disk Write Cache' on the virtual disks in windows. It is enabled by default and I read that they say when iSCSi drops and this cache is on, it can cause problems in windows. Let me know if you have any suggestions. I will keep you posted. Thanks! - Gordon

[merryweathertech]

Hi - I did want to mention that the Windows VM is 2008 R2. I just performed the same tests above but with the Windows drive write caching turned off, and the same corruption occurred. Once thing I am noticing is that when I disable the networks on the node 2 and then I copy a large file to the windows guest (around 3 GB), and then I enable the networks, it almost immediately says it is fully synchronized with node 1. Sometimes it performs a sync, and others it just says it is sync'd immediately. I am not sure how that could be, so maybe some of the file changes in the update and file copy operations are getting cached. I have this disk set as Write-Through, so I know there is some caching going on, but why wouldn't it fully synchronize each time I enable the network adapters? Thanks - Gordon

[merryweathertech]

Hi - Just performed the same thing all over again but with 'no caching' on the StarWind disk. Same corruption occurs. So I am guessing that something is causing the nodes to think they are in sync - split brain. Is it because both adapters are not going off at the exactly same time? I am disabling them one after the other, so maybe a 1 second difference, if that. Or could there be something going on with the ESXi configuration and it keeping things in memory and not committing to disk? Thanks - Gordon

[Oleg(staff)]

Could you please try to disable Jumbo frames?

[merryweathertech]

Sure - for both iSCSI and Sync? Thanks -Gordon

[Oleg(staff)]

Yes, and please share your findings after these changes.

[merryweathertech]

Hello - I eliminated the Jumbo frames and went back to 1514 MTU size on all NICs including ESXi vswitch and the same corruption occurred. Once thing I did not mention earlier is that I have the management interface on a separate 1 GB nic (main Windows NIC/IP). So three NICs in all - 1 GB Management, 10 GB iSCSI, and 10 GB Sync. When I was testing, I was only disabling the ISCSI and SYNC interfaces. When I did this, and then copied up 3 GB files to the active node, and then re-enabled the ISCSI and Sync NICS, StarWind would show the secondary node as synced immediately - which can't be right. I performed one more test all over again (with Jumbo off) but this time I disabled all three NICs on the secondary, copied up large files (which would have hit the active primary via its ISCSI nic), and then enabled them on secondary. I did this about 5 separate times and every time, the secondary node performed a Fast synchronization and there is no corruption. This was not happening before with just the two NICs disabled. So I assume that if the management NIC is still active, something is causing the dead node to think it is synced when the ISCSI and Sync NICS come back up, which is incorrect. So two questions - is this intended behavior? And second question - since each NIC is on a separate network adapter, I am afraid that if one or two fail (or related switches), can I prevent this situation from happening in that case in some way? I feel like we are making progress towards a resolution. Thanks - Gordon

[Boris (staff)]

From the information you have provided, it looks pretty much like you have not enabled any additional heartbeat link on the management interface (or any other available one) except for the iSCSI and the Sync link that you say you disable. In this case, when no heartbeat link is there, you are highly likely to get a data corruption as a result of the split brain issue. To avoid this, you need to keep an additional heartbeat link configured. As I mentioned at the beginning of this post, that can be either your management or any other interface that your nodes use.

[merryweathertech]

So if I understand correctly, ALL available interfaces on each node should have at least a heartbeat enabled for each replica? Would this take care of split brain issue in the event that any one or more interfaces drop out? Thanks! - Gordon

[Boris (staff)]

Gordon,

If you have iSCSI and Sync are configured, let's say, on the same dual port NIC and the physical NIC fails, you will need at least one more physical NIC with the heartbeat feature enabled to avoid split brain. Usually it is the management link, as most configs use management on the built-in dual- or quadport network cards, which are independent from the data ones and are not really likely to fail at the same time as the data network cards do. Ideally, all network interfaces can be configured for StarWind heartbeat, but you'd better keep that number within reasonable limits. I am not sure the whole thing has been tested in scenarios with something like 8 or 10 links used for that purpose. 3 to 5 links is something our clients use for StarWind heartbeat in the majority of cases depending on their hardware configurations.

[merryweathertech]

Hi - that resolved the issue! Now that all interfaces are part of the replica's node interfaces, no corruption occurs everything works as expected. I tried disabling one, two or all three and everything works. Rebooting the nodes work as well. I will go back to testing using Jumbo frames now. Thanks for all the info and help. I think this setup is an important thing to note in your white papers. maybe you have it but I missed it - if so can you point me to where I missed it? Thank you so much for the excellent product and support! - Gordon

[Oleg(staff)]

We recommend using additional heartbeat channels. You can find the recommendations in StarWind best practices, Heartbeat channel recommendations.