Correct procedure after planned downtime

robnicholson · Sat Aug 02, 2014 11:30 am

I've shutdown my lab one node at a time yesterday: first SAN90 and then SAN91. I've powered them back up today and all three storage devices are shown as not synchronised.

Question #1 - is this by design? I thought auto-synchronisation would have kicked in? So in our procedures, we have to document manually re-synchronising the nodes in case both nodes were shutdown - as could easily happen in a power outage. The UPS will send out the signal and everything shuts down cleanly.

I did notice that the node that was flagged as having the master copy was SAN91 which was the node that was powered down last so that makes sense.

Question #2 - will this re-sync have been a re-sync of just differences or an full re-sync? I infer the former as it finished the re-sync pretty quickly. Or do the rules that were mentioned elsewhere a few days ago kick in, i.e. it would do a full re-sync in some circumstances.

Question #3 - what state are the MPIO iSCSI connections in whilst in the un-sync'd state? There are MPIO iSCSI connections to both nodes - where do the writes go before one carries out the manual sync?

Cheers, Rob.

robnicholson · Sat Aug 02, 2014 4:32 pm

Hmm, that's strange. Gone through the same process (shut node #1 down, shut node #2 down, bring up node #1, bring up node #2) and it resynchronised automatically this time.

Mon Aug 04, 2014 9:14 am

Question #1 - is this by design? I thought auto-synchronisation would have kicked in? So in our procedures, we have to document manually re-synchronising the nodes in case both nodes were shutdown - as could easily happen in a power outage. The UPS will send out the signal and everything shuts down cleanly.

I did notice that the node that was flagged as having the master copy was SAN91 which was the node that was powered down last so that makes sense.

I`m not sure what exactly went wrong here. Could you please doublecheck if everything been done according to our HA Maintenance and Configuration Changes Guide http://www.starwindsoftware.com/ha-main ... on-changes

Question #2 - will this re-sync have been a re-sync of just differences or an full re-sync? I infer the former as it finished the re-sync pretty quickly. Or do the rules that were mentioned elsewhere a few days ago kick in, i.e. it would do a full re-sync in some circumstances.

Depends on the version - v6 assumes full sync, v8 assumes difference sync in most of the cases.

Question #3 - what state are the MPIO iSCSI connections in whilst in the un-sync'd state? There are MPIO iSCSI connections to both nodes - where do the writes go before one carries out the manual sync?

The node that is the source of synchronization should be the only node that takes care of the reads and writes from client machine. I hope that answers your question.

robnicholson · Mon Aug 04, 2014 9:40 am

I tried the same process a few more times (powering down both nodes one after the other), and it automatically resynchronised each time.

On the subject of the HA documentation, there were no targets connected so the topics on prolonged downtime don't really apply. I'll carry on doing a few more tests this week.

Cheers, Rob.

Mon Aug 04, 2014 10:56 am

OK, I`ve just got an update from our developers. Looks like I lied a bit, sorry

The autosync will kick in if the nodes were shut down correctly and the cache was flushed to disk, and there were no any write errors to disk. In all other cases nodes will be marked as not synchronized.

I hope that makes sense.

robnicholson · Mon Aug 04, 2014 11:16 am

there were no any write errors to disk

That makes sense as in the lab, the server that's using the SAN iSCSI was paused at the time of the tests. I'll do some more tests with it powered up.

Cheers, Rob.

robnicholson · Mon Aug 04, 2014 1:53 pm

Hmm, getting variable results on this. No other iSCSI targets are connected except those for HA, i.e. the server using iSCSI to the SAN is shutdown. Two tests today of shutdown node #1, then node #2, power-up node #1 and then node #2 - both required a manual re-sync operation.

epalombizio · Mon Aug 04, 2014 7:52 pm

It would really be good if there was an option to determine the last HA member to write to disk and auto sync based on that member being the primary.

While I haven't installed v8 yet, I've been bitten by this issue multiple times in v6. If we have a power outage and both HA members shutdown cleanly, there is no reason that I know of why it couldn't be setup to auto sync. I was hoping this would have been fixed in v8.

robnicholson · Tue Aug 05, 2014 1:06 pm

I agree as it's a little vague which node is the correct source. I can also confirm that auto-sync is not starting after a clean shutdown and you have to manually re-sync the nodes. To confirm steps:

No iSCSI targets connected, i.e. not IO to the SAN
Shut down SAN91
Shut down SAN90
Power up SAN90
Power up SAN91

Storage is not synchronised. If you start sync on SAN90, it syncs from SAN91. If you start sync on SAN91, it syncs from SAN90.

This is pretty much repeatable every time. At present, it we're writing the documentation for the procedure for a clean shut down, then we'll have to document manual re-sync.

Cheers, Rob.

robnicholson · Tue Aug 05, 2014 1:08 pm

I'm also a little concerned where disk IO goes when you power-up back up after a clean shutdown. The iSCSI connection is to both nodes but the nodes are not connected. What is the algorithm in this case? In the case of a power outage with clean shutdown, when power is restored and the SAN is restarted, disk IO will start. Where does it go? Does it go anywhere at all? I feel another test coming on.

Cheers, Rob.

robnicholson · Tue Aug 05, 2014 1:10 pm

Slightly more worrying is that I'm sure I've just seen a report of "full synchronisation". I'm certainly watching the LSFS storage doing a fast synchronisation. Will do some more tests.

Cheers, Rob.

robnicholson · Tue Aug 05, 2014 1:17 pm

And despite a clean shutdown and the LSFS drive supposedly doing fast synchronisation, it's taking a lot time to sync. I'm tempted to raise a support call on this specific case.

Cheers, Rob.

robnicholson · Tue Aug 05, 2014 2:14 pm

Here's another question. Two-node HA. Clean shutdown of both nodes SAN90 and SAN91 and the TEST90, the Windows 2012 server with iSCSI connections to both nodes. Assume this was a power outage situation where the UPS shut everything down cleanly.

Power restored but assume something has gone wrong with SAN90 - it hasn't powered up for some reason. SAN91 has come back online as well as TEST90. However, iSCSI is stuck in reconnecting (see attached screenshot).

At present, our entire infrastructure is down! Assume we can't get SAN90 up and running quickly (maybe we did a Windows update and it went horribly wrong). How do we bring the SAN back online cleanly on just SAN91?

robnicholson · Tue Aug 05, 2014 2:17 pm

I've just found "Mark as synchronised" which I assume is the option to be chosen in the disaster scenario? It says it'll start processing client requests immediately - sounds like that's what I want it to do. The iSCSI initiator on TEST90 is still saying "Reconnecting" - will leave it a little longer to see if it manages to reconnect automatically.

robnicholson · Tue Aug 05, 2014 2:19 pm

I do concur with epalombizio above - it would be very comforting to be able to find out which node was shutdown last/last the real up to date copy. It feels a bit hit and miss at the moment. Cheers, Rob.