[Termination tools] Contest Entry, Organisation

[H. Zantema]

Johannes Waldmann wrote:

> Will you at least re-name the files randomly for the competition?
> Otherwise it's just too easy ...

I think it is time now for further discussion on the rules of the
competition. 

The input for the competition should NOT be the data base as it is
available. 

If this is done I do not see why to organize such a competition at all:
every participant can try the examples of the database at home and may
claim which examples of this data base can be solved and which not,
independent on whether this competition is held.

It is great to have this data base, and every (potential) participant
will use this data base for testing his tool. 

If every participant submits a version of his tool in the given format
to the organizers, and the only thing what is done then is that the 
organizers again apply this tool to the same database, then the 
outcome will be the same, and the whole competition is extremely boring.

Moreover this will stimulate the development of tools that do mainly
recognition of standard examples. Most of the tools are freely available,
so every one can try these tools on the standard examples and write
a tool which only checks whether the example is in the data base and
then generate the right answer.

Another problem is that it is not fair that the moderators of the 
data base may participate themselves. In the choice about which 
examples are included and which are not they may be influenced by
what their own tools can. Accepting all examples ever submitted for the 
data base is no solution: as I said before it is easy to randomly generate
thousands of examples, and then check by which tools they can be solved.
If I want to promote some particular tool, I simply submit hundreds of 
such examples to the data base that can be dealt with the particular
tool and not by some other tools.

The only solution I see (which was proposed by other people in this 
mailing list before) is that every participant in every category
may submit a fixed number of examples, and the competition is on the
submitted examples. This has a great number of advantages:

* Now the competition is fair: neither the organizers nor the data base
  moderators do have any advantage.

* Participants are encouraged to look for `non-standard' examples. I 
  think from a scientific point of view this is much better than the
  encouragement to pattern recognition as sketched above.

* After the competition all submitted examples may be put in the data 
  base, which is ensured to be enriched in this way every year.

A number of things have to be organized well:

* A standard format for each category has to be decided. This was already
  done, so this is no problem.

* For each category some bounds have to be decided. For instance, in some
  application areas TRSs emerge consisting of hundreds of rules, while
  termination is tivial to prove. It has to be decided whether this
  kind of examples is intended to be submitted or not. I can imagine not
  to do: it would not be really fair if a participant would loose the
  competition because his very strong tool can not deal with hundreds 
  of rules. May be this should be a special category. I think it is 
  good to require some bounds on numbers of rules, size of the
  signature, size of the rules, and may be more. For string rewriting 
  I would propose an upper bound of 10 for all of these, but this may
  subject of discussion. Anyhow, these bounds should be decided for 
  every category.

* These rules should be decided and announced at least a few weeks before 
  the competition: participants should have reasonable time to develop 
  nice examples to be submitted.


 
Best regards,

		Hans Zantema.

+--------------------------------------+-----------------------------+
|                                      |                             |
|   Dr Hans Zantema                    |   Hoofdgebouw   kamer 6.73  |
|   Faculteit Wiskunde en Informatica  |   tel (040)2472749          |
|   Technische Universiteit Eindhoven  |   fax (040)2468508          |
|   Postbus 513     5600 MB Eindhoven  |   e-mail  H.Zantema at tue.nl  |
|   The Netherlands                    |   www.win.tue.nl/~hzantema  |
|                                      |                             |
+--------------------------------------+-----------------------------+