Workshop on Experimental Methods for the Assessment of Computational Systems

September 11, 2010
Krakow, Poland

Held in conjunction with the
International Conference on Parallel Problem Solving From Nature (PPSN 2010)

Workshop proceedings available as Technical Report TR10-07 at the Algorithm Engineering Group of TU Dortmund.

Scope of the Workshop

The experimental analysis of computational systems inspired by nature can be made more sound and effective by the use of appropriate experimental methods. More severe requirements have been transmitted to draw objective conclusions from computational experiments, while at the same time the design and configuration of the computational systems can be improved by profitable ways of looking into the data collected. The quest for these methods is spawning a considerable amount of interdisciplinary research, mainly between the fields of computer science, artificial intelligence, optimization, statistics and machine learning.

The aim of this workshop is to collect works in this interdisciplinary area. On the one side, we look for submissions advancing the quality of experimental methods. Some types of analysis still remain peculiar to the field of computer science and require ad hoc methods. Examples are modeling algorithm behavior over time, dealing with censored data, multiobjective and dynamic environments.  On the other side, we look for works that possibly reinvestigate previously published computational systems and algorithms and extend the characterization of their empirical performances, producing novel results.

Relevant topics for this workshop are: 

+ Tuning and configuration: new methods and comparison of existing ones
+ Modeling algorithm behavior over time and applications of survival analysis
+ Learning methods for algorithm analysis and design
+ Methods for algorithm selection in algorithm portfolios
+ Generation of data sets for benchmarking
+ Novel results on empirical assessment and comparisons of known methods in old or new settings
+ Setting up and maintenance of data repositories
+ Assessment of analysis methods in terms of errors and predictive power
+ Performance assessment methods in multiobjective and dynamic environments
+ Novel ways to visualize results
+ Simulation techniques and meta-models for algorithm analysis
+ Assessment methods for competitions
+ Experiments to validate/substantiate recent advances in the theory of heuristic computations


For online parameter control or self-adaptation of optimization algorithms you may also consider the co-located Workshop on Self-tuning, self-configuring and self-generating search heuristics (Self* 2010).

Submissions

Original contributions must be in form of abstracts (4 pages) or short papers (max 10 pages). The workshop proceedings will be published as technical report in the Computer Science series of the Dortmund University. Extended versions of selected contributions from this conference will be considered for publication in a special issue of Evolutionary Computation Journal, MIT Press.

Submissions should be formatted in the PPSN format by following the instructions of LNCS series of Springer and submitted electronically as PDF to Mike Preuss.

Accepted contributions must be presented at the workshop. There will be 4 talks of 20 minutes each plus panel discussion at the end.

Important Dates

  • Deadline for manuscript submissions: 15th July, 2010
  • Notifications to authors: 10th August, 2010
  • Revised manuscripts: 20th August, 2010
  • WEMACS workshop: 11th September, 2010
  • Foreseen deadline for the ECJ special issue: 31st December, 2010

Organizing Committee

Thomas Bartz-Beielstein, Cologne University of Applied Sciences
Marco Chiarandini, University of Southern Denmark
Luís Paquete, University of Coimbra, Portugal 
Mike Preuss, TU Dortmund, Germany

Advisory Board

Mauro Birattari, Université Libre de Bruxelles
Juergen Branke, University of Warwick
Dimo Brockhoff, INRIA Saclay Île-de-France
Gusz Eiben, Vrije Universiteit Amsterdam
Carlos Fonseca, Universidade do Algarve
David Ginsbourger, Universität Bern
Yuri Goegebeur, Syddansk Universitet
Wolfgang Konen, Cologne University of Applied Sciences
Jörn Mehnen, Cranfield University
Boris Naujoks, TU Dortmund
Ruxandra Stoean, Universitatea din Craiova
Heike Trautmann, TU Dortmund

Program

14:00-14:10 Introduction
Mike Preuss 		Room 429, Building C-2
14:10-14:50 A Subjective Review of the State of the Art in Model-Based Parameter Tuning
Tobias Wagner 		Room 429, Building C-2
14:50-15:30 Resampling Methods in Model Validation
Bernd Bischl, Olaf Mersmann, and Heike Trautmann 		Room 429, Building C-2
16:00-16:40 A Rank Transformation Can Improve Sequential Parameter Optimization
Simon Wessing and Tobias Wagner 		Room 429, Building C-2
16:40-17:20 Optimizing Support Vector Machines for Stormwater Prediction
Patrick Koch, Wolfgang Konen, Oliver Flasch, and Thomas Bartz-Beielstein 		Room 429, Building C-2
17:20-17:30 Discussion and Closing
Mike Preuss 		Room 429, Building C-2

All presentations are 40 minutes, with around 25 minutes talk and 15 minutes of a hopefully lively discussion. This is a workshop, so please bring your questions, alternative views and comments!
Please note: We have a last-minute room change from Aula U-2 to Room 429, Building C-2!

Last modified: Sun Nov 21 18:46:35 CET 2010