Challenge Overview
In this challenge you will update Solidity Benchmark Suites for Evaluating Ethereum Virtual Machine (EVM) Code-Analysis Tools, and Code to Run them to benchmark Oyente analyzer.
PROJECT BACKGROUND
The project aims to provide up-to-date public benchmarks for EVM code-analysis tools. In the current version, Mythril and Manticore tools is benchmarked (benchmarks for the later one were added in the previous Topcoder challenge). In this challenge you will expand the suite to benchmark Oyente analyzer.
TECHNOLOGY STACK
Benchmark suite is written in Python. Generated benchmark reports are in HTML.
CODE ACCESS
All links to the repositories related to this challenge were given above. The actual work will be done in the benchmark suite repository, from the commit e69d6adebbe9c972a1df6b81041c1dfa7ae3af78.
LOGIC BEHIND THE CURRENT CODEBASE
Provided benchmark suite repository imports the actual benchmark suites (first and second) as git submodules. It also contains two Python programs that run the benchmark and generate HTML reports for each benchmark suite.
The first program runner/run.py takes a number of command-line arguments; one of them is the name of a benchmark suite. From that it reads two YAML configuration files for the benchmark. The first has information about the benchmark suite: the names of the files in the benchmarks, whether the benchmark is supposed to succeed or fail with a vulnerability, and so on. An example of such a YAML file is benchconf/Suhabe.yaml. You won't have to change that. The other YAML input configuration file is specific to the analyzer. For Mythril on the Suhabe benchmark, it is called benchconf/Suhabe-Mythril.yaml. You will need to create this configuration file for Oyente or adjust the report so that these configuration files are empty. See the YAML comments in benchconf/Suhabe-Mythril.yaml for information about how to exclude a benchmark that blows up in Oyente, or to hint that a benchmark takes a long time and thus can be skipped by setting a low timeout as an option to runner/run.py
The output of runner/run.py is a YAML file which is stored in the folder benchdata with a subfolder under that with the name of the benchmark. For example the output of run.py for the Suhabe benchmark suite will be a file called benchdata/Suhabe/Mythril.yaml.
The second program, called runner/report.py, takes the aforementioned data YAML file and creates a report from that. Currently it reads the data from a single analyzer. It needs to be extended to read in data from all analyzers.
Python module dependencies are listed in requirements.txt so you can install python modules with:
$ pip install -e requirements.txt
from the github project root.
CHALLENGE REQUIREMENTS
Update runner/run.py to be able to run benchmarks on Oyente.
Update runner/report.py to include into generated reports results for Oyente.
Do not forget to update documentation of benchmark suite as necessary.
In case of any doubts, do not hesitate to ask questions in the challenge forum.