[Termtools] Questionnaire about Star-Exec

[Johannes Waldmann]

Dear all, this is my answer to star-exec organizers.
Thanks for your input. - Johannes.

> Questionnaire for organizers of Summer 2014 competitions running on
> StarExec
> 
> 1. What is the full official name of your competition, and what are the
> names and affiliations of all the organizers?

Termination Competition 2014
( http://www.termination-portal.org/wiki/Termination_Competition_2014 )

Steering Committee

    Jürgen Giesl, RWTH Aachen, Germany
    Frederic Mesnard, Université de la Réunion, France
    Albert Rubio (chair), UPC Barcelona, Spain
    Rene Thiemann, Universität Innsbruck, Austria
    Johannes Waldmann, HTWK Leipzig, Germany

Organizing Commmittee

    Johannes Waldmann, HTWK Leipzig, Germany
    Stefan von der Krone, HTWK Leipzig, Germany

> 2. What are all the StarExec job ids for jobs that were part of your
> competition?

5373 - 5391 and 5412 - 5419

( the job ids are in the URLs in the "category" columns of
http://nfa.imn.htwk-leipzig.de/termcomp/competition/20
http://nfa.imn.htwk-leipzig.de/termcomp/competition/23 )

> 3. Are those competition jobs currently publicly visible on StarExec
> (i.e., in a public space -- you can make a space public with one of the
> actions at the bottom of the page on the space explorer)?

yes, public in space 52915 but we did not advertise the space itself
(we link to the job pages on star-exec but from there,
one does not find the containing space?
It contains too many jobs anyway.)

> 4. What were the advantages and disadvantages of using StarExec for you
> as competition organizer, in comparison with whatever alternative you
> would have utilized instead of StarExec?

advantage: powerful hardware
(compared to one compute node with 16 cores that we used before)

disadvantages:

* presentation of results on star-exec is different
from what we were used to, needed to re-implement this,
as a separate service, which requires more plumbing
to get to the actual data.

* registration process does not match what we need
(a solver wants to register for a "category"
(benchmarks in a subspace) with a specific configuration,
see http://nfa.imn.htwk-leipzig.de/termcomp/registered )

> 5. What were the most positive aspects of your experience running your
> competition on StarExec?

Powerful hardware. We ran the competition on a larger data set
and with larger time-out than previous years, and still managed
to do it "live" during a conference day.

It is not longer necessary to maintain hardware (and to some extent,
software) for each competition separately. This makes the competition
more independent of individual research groups and their funding.

> 6. What were the negative aspects?

for prover authors: (see full list at
http://www.termination-portal.org/wiki/Termination_Competition_2014_Questionnaire)

* non-obvious star-exec user interface
* old software on star-exec (makes it hard to run provers)
* arbitrary restrictions
  (forbidden charactes in solver descriptions, file name lengths)

for competition organizers:

* display, registration (see above)

* inconsistent programmatic interface (star-exec URLs that
our presetation platform uses to obtain data) (sometimes
we get CSV, sometimes XML, sometimes HTML, some service URLs
are missing altogether, e.g., for display of postprocessors)

in general: intransparent process of star-exec development.

* several seemingly unrelated channels of information and discussion
  (blog, forum, emails)

* announcements of (sometimes breaking) changes only after the fact,
  or not at all.

This is actually the thing that annoyed me most.
And it continues to annoy me, see recent examples in the forums.
Why is there not a public issue tracker? The forum is a very very
poor replacement: no reasonable way of searching for issues,
no way of connecting issues to source code, to releases.

> 7. Roughly how many questions would you estimate you fielded about
> StarExec from participants in your competition?

difficult to measure. my starexec/termcomp mail folder contains
500 (received) messages. I guess 100 of them are related to Termination
"internal affairs" (e.g., what benchmarks to use),
the rest (400) is about star-exec.

> 8. Did you use StarExec's pages for viewing job results, or did you
> create your own web display for showing the data from StarExec?

own web display
http://nfa.imn.htwk-leipzig.de/termcomp/
https://github.com/stefanvonderkrone/star-exec-presenter

reasons - related to star-exec

* manage registration (and starting of jobs)

* display results (two-dimensional, and query-able)

reasons - not related to star-exec

* plan to import data from previous competitions
  (for combined queries)

> 9. (Very) roughly what percentages of your interactions with StarExec
> for organizing the competition were through the web interface, through
> StarExecCommand, and through other browser-free interaction with the
> StarExec web server (e.g., sending POST and GET requests directly to
> StarExec URLs)?

almost exclusively using star-exec URLs

> 10. Where do you think we should focus our development efforts going
> forward?  

no clear preference.

> A few of the things we are considering are: additional
> interfaces for viewing job results (maybe similar to the 2-dimensional
> grid used in Termination; other suggestions?), 

would be nice, we (Termination) will continue to work on our "grid
display" and can contribute design ideas, but I fear the Java
implementation effort (ours uses Haskell/Yesod).

> solver pipelines (output
> from one stage becomes benchmark for input to the next stage; other
> details to be determined), 

yes we need this for certified termination, in the long run.
The present solution, run the certifier as post-processor,
only works as long as there is exactly one certifier.

> more work to make it possible to install and
> run StarExec on your own server or computer (abstract out
> security-critical details, installation documention, and abstract the
> interface to GridEngine so you can use a different third-party tool to
> run jobs on a cluster, or even just your own desktop), 

yes - but not needed as long as star-exec compute nodes are available
(need a predictable time sharing mechanism, though)

> commitment to and
> documentation of the URLs used for communicating directly with the server.

most definitely!

> 11. Do you expect you or your colleagues will likely want to run the
> next edition of your competition on StarExec?

Yes - we (Termination Competition Steering Committee) have not formally
decided, but I expect yes.

> 12. How adequate was the help you were able to get either directly from
> the StarExec team or from the forum?

star-exec team was very responsive on emails
(especially right before, and during, the competition)

sometimes I got the impression that star-exec team
did not read (all) messages in the forums, at least not in time.

see above for basic reservation about intransparent development process
and insufficient tools for managing it.

> 13. How satisfied do you perceive the participants in your competition
> were with their experience using StarExec?

difficult to measure. I did not speak to anyone in person since I was
not in Vienna. I got some generally positive emails about the
competition (after all, everything was running smoothly), and some
complaints about star-exec is hard to use for solver authors
(cited above).

> 14. Were there any recurring issues or complaints from the participants
> in your competition?

cited above: non-intuitive GUI, old software, arbitrary restrictions.

> 15. Any further comments or suggestions?

not at the time. (you could re-read all of the forum messages
and check for open issues :-)

Best regards, Johannes.