[Termtools] A proposal for the next competition

[Juergen Giesl]

Dear all,

I think the question is whether we mainly want to measure
power of whether we mainly want to measure speed. I'm in
favor of mainly measuring power (also, because the systems are written
in different programming languages and therefore, runtimes
also largely depend on the language used and not only on the
algorithms implemented). Therefore, I'm not in favor of
Joerg's proposal.

However, I agree with Claude that it might be interesting to
allow more time for really hard problems. But for this, there is
also a very easy solution: one could simply run the competition
as before (i.e., with a 1 minute timeout on each example) and
afterwards one runs all tools again on those examples that nobody
could solve, but with a higher timeout. This procedure can be repeated
several times.

By the way, it would also be possible to run the competition in the
way I proposed above, but to have two different winners: the most
powerful and the fastest tool. The most poweful tool should be the
one that solved most examples. For finding the fastest tool, one doesn't
have to do any new experiments, but simply give a high weight to
short runtimes and define a certain measure which weighs runtime against
power. Then by interpreting the results of the competition in this way,
one can also define what the "fastest" tool is.


Best Regards
Juergen