[Termtools] TermComp 2018 results

[Cynthia Kop]

Hello Akihisa,

First of all: thank you for all your effort in running the competition! 
It sounds like many hours of StarExec-wrangling, and it is amazing that 
you are willing to take it on. :)

Second... I'm afraid I'm going to add your problems (even if it's only 
another bugreport to forward to the StarExec team): there still seem to 
be some problems with this final result. At least in the higher-order 
category at 
http://54.199.109.250/competitions/Y2018/termination.php?id=30047 for 
benchmark Mixed_HO_10/counterex1.xml 
<https://www.starexec.org/starexec/secure/details/benchmark.jsp?id=7652889> 
Wanda is now listed as giving MAYBE when before it was a red box, but 
the answer that I would expect for this example is NO (as that is the 
answer I immediately get when running Wanda locally). Looking at the 
data from StarExec, 
https://www.starexec.org/starexec/secure/details/pair.jsp?id=381734242 , 
the output is shown as "not available" -- unlike the timeout cases, such 
as (I think) 
https://www.starexec.org/starexec/secure/details/pair.jsp?id=381734333 , 
where the output at least shows that the tool was started.

Cynthia.

P.S. Do we want to get away from StarExec? It may be possible to get a 
group of students to spend some time developing a new competition 
platform or adapting the pre-StarExec one.


On 07/15/2018 01:36 PM, Akihisa Yamada wrote:
> Dear all,
>
> now this page shows the TermComp final result, including demonstration 
> runs.
>
> http://group-mmm.org/termination/competitions/Y2018/
>
> About run script errors: It has been detected and reported in advance 
> to the StarExec team, but apparently they couldn't fix in time. We'll 
> not have it next year.
>
> About complexity category issue: I'm sorry for not detecting this on 
> test runs... In short, StarExec produces invalid CSV files for jobs 
> that were run after last October if output contain commas (like 
> "WORST_CASE(1,?)").
>
> The StarExec team proposed I manually fix CSV files (great idea!) but 
> star-exec-presenter's dynamism doesn't allow it in a reasonable 
> effort. So the above you see is a simple and static interface I've 
> just implemented.
>
> Best regards,
> Akihisa
> _______________________________________________
> Termtools mailing list
> Termtools at lri.fr
> http://lists.lri.fr/cgi-bin/mailman/listinfo/termtools

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lri.fr/pipermail/termtools/attachments/20180715/77923fcd/attachment.html>