[Termtools] Penalties - was: Re: About memory examples in C category

[Johannes Waldmann]

On 07/19/2015 07:51 PM, Thomas Ströder wrote:

> ... Such examples won't yield any points, 
> but can lead to a penalty

I welcome suggestions for a feasible method
to handle this on starexec(-presenter).

I think starexec has a way of declaring an "expected result"
for a benchmark, but I never used that, and I don't know their
exact semantics. I'm sure it's documented somewhere,
can someone please investigate.

NB: it would look strange to include the expected result
in the benchmark itself (or put it in a public place
where tools could access it) unless it was for a certified category.

I can (on starexec-presenter) easily flag all benchmarks
where solvers give contradictory answers. But who gets penalized, then?

I am afraid of having to implement some review/rebuttal process
(like, click on a job-pair, displayed in a table,
to flag the result as doubtful) - as this means
* putting "real" data in my DB   (so far, it's only a caching mechanism
  for what's on star-exec, except for one table that shows
  what jobs belong to what competition category -
  I could delete everything else and just re-start)
* managing user accounts (so far, there's only one account, mine)
Well, I guess I have to do this at some point.

- Johannes.