Mismatch between the repo and the ICSE paper

[skywormfol]

Hi, the authors of paper "Demystifying Exploitable Bugs in Smart Contracts", thank you for your solid work and I'm inspired by your contributions.
Here I have some minor questions maybe because some misunderstandings, please provide some help, thanks!

1. there are 102 items in your contracts folder but 104 items in your results/contest.csv file, I guess 2 contracts are missed, am I right?
2. the sum of "#High" column in results/contest.csv is consistent with the item number in results/bugs.csv, both are 492. But the number in the paper should be 462 from 113 projects, how to compromise the conflicts between (492 vs 462) and (104 vs 113)?
3. I assume the benchmarks of 54 real-world exploits are not in this repo? Do you mind to share?

Thanks!

[ZhangZhuoSJTU]

Hi Gao,

Thank you for your interest in our project. You are correct; there is indeed some data mismatch between the GitHub repo and the paper. For more informatoin, you can refer to our classification.

In response to your specific questions:

1. there are 102 items in your contracts folder but 104 items in your results/contest.csv file, I guess 2 contracts are missed, am I right?

Yes, you are right. This is because there are projects whose code is not accessible. Their bugs will be marked as O2 according to the Bug Labels.

2. the sum of "#High" column in results/contest.csv is consistent with the item number in results/bugs.csv, both are 492. But the number in the paper should be 462 from 113 projects, how to compromise the conflicts between (492 vs 462) and (104 vs 113)?

As we are currently refining our classification standards and actively developing the dataset, the dataset has changed significantly. We are also incorporating new C4 projects into the dataset, which contributes to the mismatch. If you plan to use the dataset, I suggest using the most up-to-date version. It is also important to note that the observation of the prevalence of functional bugs still remains.

You may have noticed that the repo has not been updated for a few days. This is because I am occupied with my Ph.D. defense and will resume work on the project later.

3. I assume the benchmarks of 54 real-world exploits are not in this repo? Do you mind to share?

You are correct. We have not yet released the real-world exploits, but we will do so later. If you need a larger real-world exploit dataset, I recommend DeFiHackLabs, which contains nearly all real-world exploits from recent years.