I disabled the service on Dec 28 and let the other node run for this production cluster.
I started it again on Jan 14 and monitored the log. No more complains about underlying storage were observed, so I resumed Failover Cluster Node (Hyper-V) on this node.
Kept monitoring the log since then, and the following warnings about underlying storage have been logged until now.
Code: Select all
1/20 22:06:20.084156 c2c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B82170FAB0, opcode 0x88) execution time is 3484 ms.
1/20 22:06:20.084331 c2c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B82170FA80) storageRequest(0x000000B82170FAB0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3484, ms DiffTimeCompleteINIT = 3484 ms, DiffTimeCompleteEXEC = 3484 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/23 2:56:08.134101 1394 Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B82170F1B0, opcode 0x88) execution time is 3625 ms.
1/23 2:56:08.134282 1394 HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B82170F180) storageRequest(0x000000B82170F1B0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3625, ms DiffTimeCompleteINIT = 3625 ms, DiffTimeCompleteEXEC = 3625 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/23 5:45:03.800312 c2c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B78F22E8F0, opcode 0x8A) execution time is 3609 ms.
1/23 5:45:03.800503 c2c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B78F22E8C0) storageRequest(0x000000B78F22E8F0) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x2A) ex(0x0), StorageOpCode(0x8A), StorageDiffTimeCompleteEXEC = 3609, ms DiffTimeCompleteINIT = 3609 ms, DiffTimeCompleteEXEC = 3609 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/24 13:23:34.378752 16c4 Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B81A3AE090, opcode 0x88) execution time is 3438 ms.
1/24 13:23:34.379216 16c4 HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B81A3AE060) storageRequest(0x000000B81A3AE090) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3438, ms DiffTimeCompleteINIT = 3438 ms, DiffTimeCompleteEXEC = 3438 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv3'
1/27 9:04:08.423325 193c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B81D640700, opcode 0x88) execution time is 3688 ms.
1/27 9:04:08.423491 193c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B81D6406D0) storageRequest(0x000000B81D640700) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3688, ms DiffTimeCompleteINIT = 3688 ms, DiffTimeCompleteEXEC = 3688 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/27 9:04:08.424372 193c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000B81D63E600, opcode 0x88) execution time is 3688 ms.
1/27 9:04:08.424417 193c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000B81D63E5D0) storageRequest(0x000000B81D63E600) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3688, ms DiffTimeCompleteINIT = 3688 ms, DiffTimeCompleteEXEC = 3688 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
---------- Windows update, rebooted node ----------
1/28 3:49:34.952325 1e1c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000929AA47B70, opcode 0x88) execution time is 3500 ms.
1/28 3:49:34.952485 1e1c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000929AA47B40) storageRequest(0x000000929AA47B70) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3500, ms DiffTimeCompleteINIT = 3500 ms, DiffTimeCompleteEXEC = 3500 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/28 3:49:34.959652 1e1c Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000929EDF3440, opcode 0x88) execution time is 3453 ms.
1/28 3:49:34.959740 1e1c HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000929EDF3410) storageRequest(0x000000929EDF3440) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3453, ms DiffTimeCompleteINIT = 3453 ms, DiffTimeCompleteEXEC = 3453 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
1/29 9:59:30.674623 1e24 Common: CStarWindStorageDevice::AsyncReadWriteCompleted: Underlying storage request(0x000000929AB40310, opcode 0x88) execution time is 3391 ms.
1/29 9:59:30.674784 1e24 HA: Ssc_Request_Task::complete: Warning(Underlying storage): sscRequestTask(0x000000929AB402E0) storageRequest(0x000000929AB40310) GeneralFunction(Execute SCSI Command) GeneralOpCode(0x28) ex(0x0), StorageOpCode(0x88), StorageDiffTimeCompleteEXEC = 3391, ms DiffTimeCompleteINIT = 3391 ms, DiffTimeCompleteEXEC = 3391 ms. Current target: 'iqn.2008-08.com.starwindsoftware:engg-cloudhost1.enggoffice.hku.hk-csv2'
Although there are still occasional warnings of around 3 seconds each, I accepted this node is usable for production. Yet, I turned on the email notification on the Starwind so that if there are more than 10 seconds delay, I got notified.