[Termtools] When do we define how to evaluate competition results?

[Johannes Waldmann]

Talking about bylaws, the main unsolved problem (that we will be facing
in about 1 .. 2 weeks) is the evaluation of results.

There was some agreement that the Committee defines the Categories,
and this definition includes problem selection and result evaluation.

We chose to select all problems, for good reasons,
but that gives even more room for interpreting the data.

The question now is - do we want to have this discussion
(ultimately: what formula will be used to compute the winner)
before we start the competition or after?

if "before":

* need time for discussion (need to postpone competition)
* no data available (for testing the evaluation)

* discussion on evaluation is impartial
* tool authors would know what are the "important" problems,
  so they have a target for optimization

if "after":

* can start competition now (or very soon)
* real data will be available to test evaluation

* discussion is partial (we would know what formula favours what tool)
* tool authors didn't know the "importance" of problems when submitting

I suggest we go for "start now, discuss evaluation later"
(that's also what I infer from there not being much discussion on
evaluation recently).

Best regards, Johannes.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 257 bytes
Desc: OpenPGP digital signature
Url : http://lists.lri.fr/pipermail/termtools/attachments/20081030/ac5c7907/attachment.pgp