Dedup on single core?

[TstAgs]

Hi,

we are testing the Starwind iSCSI solution, with a virtual deduplicated device.

We've noted that on a 8 cores machines, just 1 core is used during the deduplication process.

Can someone confirm that the Dedup engine just use a single core?

We are searching for a Deduplication Multicore solution in order the make Deduplicated Backups.

Thank you very much

[anton (staff)]

1) It does not use single core. StarWind is multithreaded so many workers execute the I/O jobs.

2) You need RESULT and not TECHNOLOGY. So if dedupe coeff. are fine and hardware resources usage is fair and app is stable I don't think you should care how it's written, designed and so on

Hi,

we are testing the Starwind iSCSI solution, with a virtual deduplicated device.

We've noted that on a 8 cores machines, just 1 core is used during the deduplication process.

Can someone confirm that the Dedup engine just use a single core?

We are searching for a Deduplication Multicore solution in order the make Deduplicated Backups.

Thank you very much

[TstAgs]

Hi,

thanks for the reply, i was asking because with a Dedup virtual drive we can reach a max performance of 40 MBytes / Sec, with 80% allocated ram on the Starwind server, and just 1 core using 15% CPU.

The other CPUs are 0 % used, so it's clearly using a Single core for the dedup process.

I can give you another information. In the task Manager i can read the the Dedup process is using 20 Threads.

How can i fix this?

Thank you very much

[TstAgs]

I allegate 2 screenshots showing the situation

Clipboard01.png (56.76 KiB) Viewed 5542 times

Clipboard02.png (17.58 KiB) Viewed 5542 times

[TstAgs]

I tried to add a second Dedup Virtual Drive and copying 3 files concurrently on the 2 drives.

The situation doesn't change and the overral throughput from the Client to to the iSCSI target remains of 30 Meg / Sec with a CPU % of 12 on the Starwind server

Clipboard03.png (17.64 KiB) Viewed 5547 times

[anton (staff)]

CPU is not limiting factor with StarWind dedupe: disk subsystem is. So giving more cores / GHz is not going to improve anything as
hash caclulations on critical path is maybe 5% of time (maybe less). Throwing more spindles / going flash DOES help to boost performance because
of storing metadata stealing IOPS.

Vitaly would be able to tell more tomorrow.

[Vitalii (staff)]

That version does not support multi-threading.

But, the deduplication library supports using multiple threads. It will be available in the next major release.

[TstAgs]

@anton
We were aware of the storage requirements so we put the Dedup repository on a 15k FC San.
Moreover, as you can see from the graph, disk occupation is far below a single SATA disk bandwidth.

@Vitalii
Thanks for the reply. So we'll wait for the next major version in order to retest the Dedup again with Multi core support.

Thanks

[Anatoly (staff)]

Thank you very much for your feedback!
We are really glad that your issue get solved, and we truly hope that none will appear again.
It was our pleasure to assist you.