Hash configuration information for the dedup performance test of DataJuicer 2.0

[cist]

Before Asking 在提问之前

• I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

• I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

• I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

hi, Do you have the MinHash configuration for this deduplication performance test? How many hashes were calculated and how many bands were used?

Additional 额外信息

No response

[chenyushuo]

Hi @cist , thanks for your attention and using on Data-Juicer and our paper!

In our experiment, 256 hashes were calculated and 32 bands were used.

[cist]

Is there detailed physical configuration information for this test data? Such as memory, disk, CPU information and so on? The test results in our own cluster are much slower.

[Cathy0908]

Hi @cist we used the PAI (https://help.aliyun.com/zh/pai/) cluster and the ray framework they support on PAI DLC. A single node has 160 vCPU and 1800 GiB. The data was stored on CPFS with a bandwidth of 12 GB/s.