Restoring Tapes from Azure Blob Archive tier storage

dgbegfb · Fri Dec 15, 2023 6:28 pm

VTL tapes are not immutable by design. So, is your request about introducing immutability on the VTL level?

Not exactly, I can and I do a tape immutability at Azure level using retention policy but VTL doesn't see it and doesn't check so when VTL issue a deletion command it doesn't report about an unsuccessful operation.

Yes, please I will be happy to have that chart. Thanks!

Here you are. The first is upload/download by VTL and the second one is the same thing using web gui.

: Screenshot 2023-11-06 at 22.44.54.png (152.64 KiB) Viewed 55025 times

: Screenshot 2023-11-06 at 22.47.32.png (230.58 KiB) Viewed 55025 times

Tue Dec 19, 2023 7:15 pm

I will forward your request to R&D Stay tuned!
Thanks.

Fri Sep 27, 2024 11:36 pm

Hi,

I finally got some updates.
1. CloudReplicator does not support immutable storage yet. It was put on the road map.
2. It is possible to increase the block size in replicator settings to reduce the number of i/o requests.

dgbegfb · Wed Dec 11, 2024 4:13 pm

Hi.

Thank you for the feedback.

1. I don't know what do you mean then talk about support of immutable storage. I meant just a deletion errors handling for whatever reason they arise whether because of a storage immutability or any other reason.

2. There no such variable in VTLReplicationSetting.ps. Do you mean that I can edit config.json in CloudReplicator folder? That the value of AzurePartSizeMb is safe, optimal or recommended?

3. You still have provide with no feedback to another topic from my previous post:

VTL rehydrate archive tape using Set Blob Tier to the same container. It leads to early deletion and impossibility of using life cycle management in the case of immutability. Microsoft recommends using Copy Blob command to Hot tier and if it can be done to another container it will be possible to delete it automatically.

Is this issue on the road map or it's just an out of importance detail for you?

Wed Dec 11, 2024 4:50 pm

You are always welcome

1. I don't know what do you mean then talk about support of immutable storage. I meant just a deletion errors handling for whatever reason they arise whether because of a storage immutability or any other reason.

I was answering your question about immutability support.

2. There no such variable in VTLReplicationSetting.ps. Do you mean that I can edit config.json in CloudReplicator folder? That the value of AzurePartSizeMb is safe, optimal or recommended?

You can tweak the Cloud Replicator JSON, however, I am not sure if adding immutability support as a parameter there will help.

3. You still have provide with no feedback to another topic from my previous post:

I believe it is on the Road Map. I have brought this thread to the dev team, so they should be aware of your suggestions.

many thanks for your efforts!

dgbegfb · Wed Dec 11, 2024 10:47 pm

Thanks, everything is clear but I have a mess with one question:

I asked previously:

using VTL to upload/download tape to/from Azure it uses much many write/read operation in comparison to make the same operation by web gui. I have a chart to compare it if you will need. It just the question of money.

You answered:

It is possible to increase the block size in replicator settings to reduce the number of i/o requests.

I asked then how to do it if there is no such option in VTLReplicationSetting.ps?
and your answer a bit irrelevant:

You can tweak the Cloud Replicator JSON, however, I am not sure if adding immutability support as a parameter there will help.

so how to increase the block size in replicator settings in a right way?

Wed Dec 11, 2024 11:55 pm

You are always welcome. Sorry if my answers appear to be misleading.
Yes, you are right.

dgbegfb · Wed May 14, 2025 12:08 am

Hi.

As we discussed previously I have changed "AzurePartSizeMb": "1024" in StarWind\CloudReplicator\config.json and have got an issue during replication.

The replication started but has never finished. As I can see in the log it try to operate by a different part size "Calculated desired part size: 100000000".

More over right now I have a lot of stuck replication jobs (see a picture).

: Screenshot 2025-05-14 at 01.48.40.png (24.06 KiB) Viewed 12721 times

So I have two questions:

- how to cancel the stuck jobs ?

- how to properly increase a block size ?

Thanks.

Wed May 14, 2025 8:03 am

The size is imposed during tapes' creation, and you can split it in the config.json file.
Try restarting StarWindService.

dgbegfb · Wed May 14, 2025 8:34 am

yaroslav (staff) wrote: ↑
Wed May 14, 2025 8:03 am
The size is imposed during tapes' creation, and you can split it in the config.json file.
Try restarting StarWindService.

Dear Yaroslav,

but it was your advise to increase the block size at config.json file in order to decrease the number of put blob operations. Please see the previous posts. I still need to decrease them as Starwind splits each tape for 10MB blocks.
Restarting of starwind services doesn't hepl unfortunately. It should be any internal way to cancel a replication job.

Wed May 14, 2025 8:38 am

There is a recommended size for the upload to the cloud (e.g., 4 GB).
Tweaking the JSON is about finding the balance.

dgbegfb · Wed May 14, 2025 8:53 am

Have you tried to read and understand my questions? I guess no becase your last answers is out of relevance. I really need help please try to read first.

Once again I have two urgent question and the others may faced with the same:

1. How to cancel the stuck replication jobs?

2. I have tapes the size of that under a full control by the PS scripts. Let's say they are of 50GB each. During cloud replication each tape splits for 10MB blocks that generates a lot of blob operations at a cloud level. At the same time uploding such tape by UI takes only one blob operation. That's why I asked about a solution to increase a block size and you advised to do it at config.json. I did the block size of 1024MB and it doesn't work. So what sould I do next?

Wed May 14, 2025 9:15 am

1. How to cancel the stuck replication jobs?

Try restarting the service.

2. I have tapes the size of that under a full control by the PS scripts. Let's say they are of 50GB each. During cloud replication each tape splits for 10MB blocks that generates a lot of blob operations at a cloud level. At the same time uploding such tape by UI takes only one blob operation. That's why I asked about a solution to increase a block size and you advised to do it at config.json. I did the block size of 1024MB and it doesn't work. So what sould I do next?

The tape upload takes time. What you see in UI is the tape uploading, not its chunk.
Find the balance between upload size and the number of operations. I can't advise on that as it is a purely empirical process.
What I see people do is split tapes into sub-4GB blocks.

dgbegfb · Wed May 14, 2025 9:40 am

Yaroslav, are you an AI-agent?

Try restarting the service.

I wrote I tried and it doesn"t work!

The tape upload takes time. What you see in UI is the tape uploading, not its chunk.
Find the balance between upload size and the number of operations. I can't advise on that as it is a purely empirical process.
What I see people do is split tapes into sub-4GB blocks.

UI I meant UI of Azure cloud web UI. How I can find balans?! Where should I find?! Setting it in config.json breaks a replication job.

Wed May 14, 2025 10:34 am

You are speaking to a living person who is trying to help, so I would appreciate some respect.

How I can find balans?!

I can't advise on that, as it is an empirical process. Setting too small results in too many operations too large might be not optimal for upload.
What I see people do is split tapes into sub-4GB blocks.

Setting it in config.json breaks a replication job

Isn't stoping the job something you would like to achieve?