[Termtools] A proposal for the next competition

[Jörg Endrullis]

Juergen Giesl wrote:
> However, I agree with Claude that it might be interesting to
> allow more time for really hard problems. But for this, there is
> also a very easy solution: one could simply run the competition
> as before (i.e., with a 1 minute timeout on each example) and
> afterwards one runs all tools again on those examples that nobody
> could solve, but with a higher timeout. This procedure can be repeated
> several times.

I am not in favour of this proposal. Consider the following scenario: we
have two tools {1,2}, 100 problems. Tool 1 solves all problems and needs
50 seconds for each of them. Tool 2 solves the first 99 problems in 1
second, but fails for the last problem since it would need 61 seconds.
Which of them is the better tool?

I think we should take the most natural role model, namely students
writing an exam. Clearly you have a time-out for the whole exam and it
is up to the students how to spend the time on the individual tasks. I
think everybody would agree, that it would be unfair to write an exam
with a per task time-out? Every student has different strengths and its
the same for our tools.

Kind regards,
Jörg Endrullis