David R. Cok, Aaron Stump, and Tjark Weber. Technical Report 2014-017, Department of Information Technology, Uppsala University, July 2014.
After 8 years of SMT Competitions, the SMT Steering Committee decided, for 2013, to sponsor an evaluation of the status of SMT benchmarks and solvers, rather than another competition. This report summarizes the results of the evaluation, conducted by the authors. The key observations are that (1) the competition results are quite sensitive to randomness and (2) the most significant need for the future is assessment and improvement of benchmarks in the light of SMT applications. The evaluation also measured competitiveness of solvers, general coverage of solvers, logics, and benchmarks, and degree of repeatability of measurements and competitions.
@techreport{cok14the,
author = {Cok, David R. and Aaron Stump and Tjark Weber},
title = {The 2013 {SMT} Evaluation},
number = {2014-017},
institution = {Department of Information Technology, Uppsala University},
department = {Computing Science Division},
month = jul,
year = {2014},
url = {http://www.it.uu.se/research/publications/reports/2014-017/},
}
A revised version of this paper is available.