Large Data 3-Way Synchronous Mirrors

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

Post Reply
CRD_MWCIT
Posts: 5
Joined: Mon Jan 18, 2021 4:20 pm

Mon Jan 18, 2021 5:36 pm

Hello everyone!

My team and I are having trouble getting a 3-Way Synchronous mirror setup on our new storage enclosures from Dell. We are feeding the iSCSI storage to 4x vmware vsphere 7.0U1 hosts

We are currently stable with 2-Way Sync. When we attempt a full-sync of the 3rd SAN we immediately begin experiencing performance issues on the sync partner. We also begin receive error logs indicating underlying storage taking longer than expected and partner node taking longer than expected to respond. For example

"command "WRITE". Request execution time is longer than expected. Response time is 13 sec."

The 3 storage nodes consist of the following hardware

The Starwind host is a Dell Poweredge r740xd with Windows Server 2016 Datacenter
The storage enclosure is a Dell Powervault ME4084 with 56x 12TB 7200RPM SAS drives
The iSCSI Network is 2x 10GbE Mellanox ConnectX-4 LX and Dell S4048-ON switches
The Sync Network is 2x 40GbE Mellanox ConnectX-4 and Dell S6010-ON switches

I have attached a diagram of the hardware setup.

Any help would be greatly appreciated, I am happy to send more information and system logs if needed
Starwind Big Data.png
Starwind Big Data.png (117.83 KiB) Viewed 3368 times
Replication Node Interfaces.png
Replication Node Interfaces.png (14.45 KiB) Viewed 3368 times
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Mon Jan 18, 2021 8:29 pm

Hi,

Welcome to StarWind forum. Can I have the logs from your system, please? What RAID level do you use?
Please note that delays during full synchronization are expected as full synchronization is a resource consuming process. Use this tool to collect the logs from all storage boxes https://knowledgebase.starwindsoftware. ... collector/. Share them via OneDrive or Google Disk.

In your description, you mentioned only one server, but you are discussing a 2-way mirror here. Could you pelase tell me how many storage servers do you have?

Please use IPs from different subnets for iSCSI.
CRD_MWCIT
Posts: 5
Joined: Mon Jan 18, 2021 4:20 pm

Mon Jan 18, 2021 9:04 pm

Thank you Yaroslav,

We have 3 nodes, all nodes have the exact same hardware

3x Dell Poweredge r740xd with Windows Server 2016 Datacenter
3x Dell Powervault ME4084 with 56x 12TB 7200RPM SAS drives

I will split the A and B iSCSI channels into 192.168.95.x and 192.168.96.x subnets and I will gather logs for you
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Mon Jan 18, 2021 9:08 pm

Greetings
What is the RAID level used for storage boxes?
CRD_MWCIT
Posts: 5
Joined: Mon Jan 18, 2021 4:20 pm

Tue Jan 19, 2021 4:37 pm

I have attached a clearer diagram. I still need to change iSCSI to separate subnets but this will require time since we have production data on the storage already
Starwind 3-Way.png
Starwind 3-Way.png (207.79 KiB) Viewed 3326 times
yaroslav (staff)
Staff
Posts: 2277
Joined: Mon Nov 18, 2019 11:11 am

Thu Jan 21, 2021 9:22 pm

Greetings,

I noticed the delays on the first server will be investigating that further.
Post Reply