[Termination tools] Rules of the WST competition

[H. Zantema]

On Tue, 27 Apr 2004, Claude Marche wrote:

> >>>>> "H" == H Zantema <hzantema at win.tue.nl> writes:
> 
>     H> I think a main goal of the competition is to be able to compare the 
>     H> power of various tools in a more objective way than by reading papers
>     H> about these tools. I am afraid that this desired objectivity is
>     H> hard to reach if
> 
>     H> * the competition is only on systems from some data base while 
>     H>   the moderators of this database are participants too, and
> 
> I'm not a moderator of the database, I included every problem that
> anyone submitted. I contributed myself for only a very small part of
> the TPDB. I really don't understand your doubts about my objectivity.

I do not have any doubts about your objectivity. But as you said, it is
expected that some maintenance is expected to be done (like removing
the double occurrence of various/20.trs = SRS/z1.srs), and there can
be reasons for not including systems. For every outsider it is clear
that if the person responsible for maintenance also participates 
in the competition, there is a risk of non-objectivity. It should be
clear how to deal with this, just like for every conference it should
be clear how to deal with program committee members that want to submit
a paper to the conference. For this year we can keep the rules as you
propose, but for the future this should be discussed.
 
>     H> * there is an obvious way to perform well in the competition by a 
>     H>   simple fake program.
> 
> But a tool doing that will be "disqualified" in the sense that nobody
> will trust it anymore. Its author will have the word "cheater" on his
> face for the rest of his life.
A tool both checking a data base and trying various techniques would not
immediately be a cheater because it checks a database. It is clear that
as soon as tools of this shape come up using TPDB (what else?) then the
present format does not give interesting information any more. Again
for this year this is not expected to be a problem, but for the future
it should be discussed.


> I will make two classifications, one counting the NO and one not
> counting the NO.
Very good!

		Best regards, Hans Zantema.

+--------------------------------------+-----------------------------+
|                                      |                             |
|   Dr Hans Zantema                    |   Hoofdgebouw   kamer 6.73  |
|   Faculteit Wiskunde en Informatica  |   tel (040)2472749          |
|   Technische Universiteit Eindhoven  |   fax (040)2468508          |
|   Postbus 513     5600 MB Eindhoven  |   e-mail  H.Zantema at tue.nl  |
|   The Netherlands                    |   www.win.tue.nl/~hzantema  |
|                                      |                             |
+--------------------------------------+-----------------------------+