Corruption problems

[emachine74]

I'm testing out StarWind as an iSCSI solution for a Windows 2008 R2 cluster we're planning. Setup went well, but now that I'm trying to actually use it, I'm having excessive incidents of LUNs becoming unusable, for one reason or another, by either the nodes of the cluster, or the VMs themselves. This is under virtually no load what so ever.

Here's the setup:

The SAN

A Windows 2003 SP2 (x86) box

• It's a PDC
  It runs StarWind 4.0 (the 15-day trial)
  It has a 1TB RAID1 (2 of these in mirror)
  It has a 1Gb NIC dedicated to iSCSI

The Cluster

A Windows 2008 R2 (x64) box

• It's node 1 of 2
  It runs the StarWind VSS Support Module
  It has a 1Gb NIC dedicated to iSCSI

Windows 2008 R2 (x64) box

• It's node 2 of 2, and it's hardware & software are identical to node 1

The cluster validates 100%, and is up.

The implementation

The Quorum drive is a 150MB .img (not an .ibv). It seems to be working just fine, all the time.

There are 4, Highly-Available VMs:

• VM1
  VM2
  VM3
  VM4

These VMs do not use .vhd files, I chose instead to use pass-through disks. To accomplish this, 1 IBVolume is created for each VM - these will be the OS/boot disk(s). Another .ibv is created to store the configuration data (xml, bin) for each VM. All .ibvs are cdp/snapshot devices scheduled at max 2 over 720 minutes. In StarWind, I have 8 .ibv's that look like this:

• VM1_OS.ibv
  VM1_CFG.ibv
  VM2_OS.ibv
  VM2_CFG.ibv
  (etc.)

The Problem

VMs maintain reliability for no more than ~24 hours. This is just a sandbox environment, so I've created & destroyed a lot of LUNs and VMs over the past week. Sometimes, I can't even install the OS onto a pass through LUN at all (from within the VM, the Windows installer claims the disk can't be installed to: but if I "start over" and recreate the LUN, it will suddenly work). Of the VMs which have an OS and are running OK, eventually I will encounter a problem along the lines of:

• When migrating a running VM (either via move, or live migration) the destination node might see the configuration LUN (e.g., VM2_CFG.ibv) as "not formatted", which is erroneous.
  When restarting a VM, sometimes the VM is unable to boot from it's primary hard disk - or, it blue-screens loading Windows, etc. It's OS disk simply becomes "corrupt", as far as I can tell.
  In a VM that had Active Directory installed, 24 hours after installing AD that VM was be unable to load AD due to an "unspecified error".
  Checkdisk runs frequently when booting VMs, and seems to repair quite a lot of stuff from time to time.
  When altering the BIOS configuration for a VM, the result can be a corrupted / destroyed BCD, of course rendering the VM unable to load it's OS

There are a lot of variables at play here. But these errors tend to indicate that storing data for VMs onto my iSCSI target is producing a completely unreliable environment.

The cluster nodes themselves appear to be running well. The cluster itself is healthy, and I have never had a problem with the Quorum drive becoming unavailable, which is just another iSCSI target volume - but, it is an .img, not an .ibv.

It seems to be true that, whenever I try to use an .ibv to either store configurations for VMs, or, use an .ibv as a pass through disk for a VM, the odds of corruption over time are effectively 100%.

Has anyone had success using StarWind to operate long-running, highly-available VMs in a 2008 R2 environment with a StarWind configuration similar to this?

[Robert (staff)]

Hello,

We have seen this issue when .ibv files were copied by DiskShadow utility into its own snapshots. DS snapshots were then mounted as drives to the system/ .ibv files were accessible from these mounted snapshots, but once mounted, they were shown as "RAW and needs formatting". Once copied to another location - everything mounted just fine.

If it's not the case, please submit a request at http://www.starwindsoftware.com/support - we will investigate it and find a workaround.

Thanks
Rob

[emachine74]

when .ibv files were copied by DiskShadow utility into its own snapshots.

What would cause an .ibv file to be copied by DiskShadow utility onto it's own snapshot?

[Robert (staff)]

Hello,

DiskShadow creates its own snapshots that can be mounted as local drives. We just backed up the .ibv files and then mounted the DiskShadow snapshot with .ibv to a local drive.

Thanks
Rob