Had a crash, Resync Perforamnce Questions

[mooseracing]

We had a crash and the software decided to do a full sync instead of fast. Its been about 17hrs and our ~1.5TB of data isn't done yet.

We have 2 targets - ~1tb & 500GB, we use HA for both.

Watching perfmon on what we will call Server 2, the one that is receiving all the sync data, Avg Write queue is staying at 1. Starwind is using about 13GB's of RAM, and under 5% CPU cycles. The 10Gb sync interface is hovering at 1% utilization.


Avg Read Queue on Server 1 is showing about .01 -.05 . Perfmon is using a scale of 100.

These servers are exact duplicates except one of the arrays has a couple more drives.
Server 1
VD1 - 1TB, 8- 10k 2.5" drives in a R10 -1TB Target
VD2 - 1TB - 8 - 10k drives in a R10 ~500 GB Target

Server 2
VD1 - 1TB, 8- 10k 2.5" drives in a R10 - 1TB Target
VD2 - ~500GB - 4 - 10k drives in a R10 ~500 GB Target


I have played with the Re-sync priority, but it made no difference in making my VM servers available so I moved the slider back to the fastest possible sync.

What I don't get is why isn't more of the sync channel being used? Any of those raid sets are capable of more than 1% that the NIC is showing being utilized. The sync channel is 10Gb server to server, no switch. The HB is a direct connected Gb. Both servers connect back to our network via 10Gb through a Dell 6224.

I've moved our most important servers off the SAN temporarily since we need access now. I only have a Sharepoint server, and a couple other real lightweight stuff running.

Ideas on where to focus?

Forgot to mention we are using 5.7

[anton (staff)]

What exactly crashed? StarWind itself or any other component? Do you happen to have any logs / crash dumps?

[mooseracing]

Windows crashed, it seems like every few months this one will crash and restart. I haven't had time to look into yet.

the 500Gb Target 2 just completed syncing, utilization of the sync channel while Target 1 is syncing has now dropped to .4-.5 %


edit: Forgot to mention when this has happened before, a fast sync would complete and we would go back to life as normal. This is the first full sync from this issue.

[mooseracing]

I only have 3 light things running on our HA right now as I am still trying to get some better performance. I noticed yesterday for some reason when running different disk bench marks locally, my read speeds were correct on the raid set, but writes were down around 30-40 MB/s. I updated the Raid card (LSI 9260-8i) driver on both servers, rebooted and the write speeds were back up in 400MB range where I expected them to be.

Going back to copying files though:

Local Copying between Raid sets to the slower one I get about 200MB/s, which is about expected for reading from the smaller/slower raid set. I was planning on making the sets larger this year, but the flooding and skyrocketing drive prices put a hold on that.

Copying between the 2 SAN servers across the 10Gb network on the larger raid set I see about 200-300MB/s, Task Mgr showing around 25% utilization of the 10Gb link. Same thing but the smaller raid set about 130, utilization of around 12%.

Now going to my cluster servers - 2008 R2 w/SP1, and copying files through iSCSI using the cluster server that has ownership of the drive:

Local to Small set - About 130MB/s, 8-19% of the 10 Gb link
Local to Large Set - Same as above

I've done the tweaks in the sticky. I bumped my TX/RX Queues up on the NIC's to 1024, but that seems to cause connectivity issues under high loads, so I bumped that back to default at 512. The rest of the NIC settings are default, jumbo frames set at 9014. The sync channel is direct from server to server.

I was hoping to see at least 200MB/s, otherwise short of improving random access, adding more drives isn't netting me much.

Is there some other tweaks that would be worthwhile?

[mooseracing]

To add to this I found a couple more things I need to look at more.

Digging through the logs I see there is a couple times when restart it couldn't allocate enough RAM for the cache so it defaulted to 512MB. Seems a little odd as was giving each target 7GB for a total of 14GB of the 16GB available. These servers aren't doing anything else. I'm going to drop these to 6GB I think this weekend when I can bring it down.

Also in the logs I see this alot:
2/24 10:20:08.325 a64 PR: PREEMPT: Existing reservation type: 5.
2/24 10:20:08.325 a64 PR: No registrations preempted.!

On the other:
2/24 10:20:08.384 640 PR: PREEMPT: Existing reservation type: 5.
2/24 10:20:08.384 640 PR: Keep registration key 000076E97466734D owned by iqn.1991-05.com.microsoft:bart.dw.local,400001370001 iqn.2008-08.com.starwindsoftware:burns-witness0328
2/24 10:20:08.384 640 PR: No registrations preempted.!

Anything to be concerned about?

[Anatoly (staff)]

Well, I think the best step to move forward to solving your issue is to upgrade your StarWind. Please take a close look at the upgrade notes on the download page.

[mooseracing]

I am planning to upgrade from 5.7, not sure if I can fit in the down time yet. I have just enough room to move 90% of the targets off of the SAN and onto local Hyper V servers and still run them decently.


But to update this more, I changed my larger target's RAM allocation to 5GB's instead of 7GB, and I am not seeing any allocation errors for cache now. Also for some reason I must have typed the HB and Sync channels incorrectly on that target as I noticed one server would use the HB during Syncing. The HB channel is only Gb.

I had my large target - 1Tb, doing a compelte Full Sync in an hour over the weekend. It was utilizing around 25% of the 10Gb link. Which is roughly closer to the 300-400 MB/s (locally) that Raid set is capable of. I could have lived with that. But of course this morning I just kicked off a full sync again to see how things are going and I am down to 18% of the 10Gb. So more digging to see what I find is hampering us.

edit: Forgot to mention it was good to see Total I/O's from Starwind graphs, in the 1000-1500 range. I think everytime I have looked before we were never above 500. So there has been some headway made.

[mooseracing]

Back to square one, I know the raid sets are capable of 400MB, is my network capable?

I'm using 10Gb CX4 Intel dual port cards, Sync is direct, LAN is through a Dell 6224.
Other than Default settings:
Jumbo - 9014
RSS Q's 8
RSS Processors 16
All registry tweaks in sticky thread.

Anyones input on iperf so I can saturate the 10Gb link?

This is what I am doing:

C:\>iperf -w 4M -c 192.168.11.4
------------------------------------------------------------
Client connecting to 192.168.11.4, TCP port 5001
TCP window size: 4.00 MByte
------------------------------------------------------------
[180] local 192.168.11.5 port 52174 connected with 192.168.11.4 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0-10.0 sec 2.50 GBytes 2.14 Gbits/sec

Obviously not close to ideal, I get about 2Gb whether through the sync channel or through the switch.

Ideas?

[Max (staff)]

Ideas?

I'm not really an Iperf fan, but when I first played with it there was a set of parameters to set up a queue, which actually saturated the channel up to ~90 %
I know a workaround which actually tests both iSCSI and network performance in a little more comprehensive UI.
Use StarWind's RAM disk mounted over 10 GbE as a destination for IOmeter tests, you can start the test and dynamically tune the network being able to see which valve does the correct thing.
I usually set up IOmeter for at least 4 workers with 64 outstanding IO's, every worker is bombing the drive with 32k reads & writes.
After the TCPack and JF set to 9k I usually got at least 10xx mb/s for Writes and 11xx mb/s for reads (for Intel x520 SFP+).
Myricom cards show 10-20 mb/s more but only if you use the latest drivers and enable Adaptive Interrupt Moderation

[mooseracing]

I couldn't find what I needed for iperf, so I opened multiple windows and ran 4 instances at once. I was able to use around 75% of the 10Gb link, which is good enough for me.

I'm trying a 5GB RAM drive and IOMeter but so far network utilization is less than optimal.

[mooseracing]

So IO Meter and a RAM Disk.

100% Seq, 100% Read, 4 Workers, 64 Outstanding, 32k size
around 400MB/s

100% Seq, 100% Read, 4 Workers, 64 Outstanding, 2MB size
around 700-800MB/s

Writes seem to be about 20% slower.
---------------------------------------------------------------
Creating a CDP/Snapshot device with 3GB WB cache
I get 250MB/s write with the above 2MB settings, 440MB/s read.

Locally with the same IOMeter settings I am getting 352MB/s write and just under 600MB/s reads.

Should I be losing that much?

[Anatoly (staff)]

Not really. Have you tried to use Basic Image target in the test instead of CDP? If not could you please?

Thanks!

[Max (staff)]

Could you please check that all 4 workers have the disk and the 32k read assigned, this is one of the key factors here.

[mooseracing]

Ok, I was able to update to 5.8.1949 over the weekend.

Test 50GB basic image (2GB of WB cache), 4 workers, 64 outstanding, 32k 100% Read - Sequential

So far about 25% utilization of the 10GB. 270MB/s

Same test locally though - but about 500MB/s


Details - maybe this will help determine where the issue is

SAN
Local

IO/s
8574
15577

MB/s
268
487

Avg IO Response
29.8
16.4

Max IO Response
193
82

CPU Utilization
20 %
15%


I can't say for sure on the past test about the workers, but I triple checked this time that all 4 were assigned correctly.

[Max (staff)]

I recommend to use the RAM disk test before you make sure that the link is properly saturated.
1. Double check that the latest drivers are used
2. doublecheck JF values (sometimes those drop to 1.5k)
3. If you have more than 4 virtual cores available you can max out the # of workers to be equal to the # of virtual cores.
I would also doublecheck the chipset driver to make sure it's up to date. In a few cases this caused general performance issues.
Also please tell us what is the CPU models you're using in the SANs