EDML
2020

2nd Workshop on Evaluation and Experimental Design in Data Mining and Machine Learning (EDML 2020)

Workshop at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Database (ECML PKDD 2020), September 14‑18, 2020

Description

A vital part of proposing new machine learning and data mining approaches is evaluating them empirically to allow an assessment of their capabilities. Numerous choices go into setting up such experiments: how to choose the data, how to preprocess them (or not), potential problems associated with the selection of datasets, what other techniques to compare to (if any), what metrics to evaluate, etc. and last but not least how to present and interpret the results. Learning how to make those choices on-the-job, often by copying the evaluation protocols used in the existing literature, can easily lead to the development of problematic habits. Numerous, albeit scattered, publications have called attention to those questions [1-5] and have occasionally called into question published results, or the usability of published methods.

Those studies consider different evaluation aspects in isolation, and the issue becomes even more complex because setting up an experiment introduces additional dependencies and biases: having chosen an evaluation metric with little bias can be easily undermined choosing data that cannot appropriately treated by one of the comparison techniques, for instance, and having carefully addressed both aspects is of little worth if the statistical test chosen does not allow to assess significance.

At a time of intense discussions about a reproducibility crisis in natural, social, and life sciences, and conferences such as SIGMOD, KDD, and ECML/PKDD encouraging researchers to make their work as reproducible as possible, we therefore feel that it is important to bring researchers together, and discuss those issues on a fundamental level. In non-computational sciences, experimental design has been studied in depth, which has given rise to such principles as randomization, blocking, or factorial experiments. While these principles are usually not applied in machine learning and data mining, one desirable goal that arose during workshop discussions is that of the formulation of a checklist that quickly allows to evaluate the experiment one is about to perform, and to identify and correct weaknesses. An important starting point of any such list has to be: “What question do we want to answer?”

An issue directly related to the dataset choice mentioned above is the following: even the best-designed experiment carries only limited information if the underlying data are lacking. We therefore also want to discuss questions related to the availability of data, whether they are reliable, diverse, and whether they correspond to realistic and/or challenging problem settings. This is of particular importance because our field is at a disadvantage compared to other experimental science: whereas there, data are collected (e.g., in social sciences), or generated (e.g., in physics), we often “only” use existing data.

Finally, we want to emphasize the responsibility of the researchers to communicate their research as objectively as possible. We also want to highlight the critical role of the reviewers: The typical expectation of many reviewers seems to be that an evaluation should demonstrate that a newly proposed method is better than existing work. This can be shown on a few example datasets at most and is still not necessarily true in general. Rather it should be demonstrated in papers (and appreciated by reviewers) to show on what kind of data a new method works well, and also where it does not, and this way in which respect is different from existing work and therefore is a useful complement. A related topic is therefore also how to characterize datasets, e.g., in terms of their learning complexity [6] and how to create benchmark datasets, an essential tool for method development and assessment, adopted by other domains like computer vision, IR etc.

Topics

In this workshop, we mainly solicit contributions that discuss those questions on a fundamental level, take stock of the state-of-the-art, offer theoretical arguments, or take well-argued positions, as well as actual evaluation papers that offer new insights, e.g. question published results, or shine the spotlight on the characteristics of existing benchmark data sets.

• Benchmark datasets for data mining tasks: are they diverse/realistic/challenging?
• Impact of data quality (redundancy, errors, noise, bias, imbalance, ...) on qualitative evaluation
• Propagation/amplification of data quality issues on the data mining results (also interplay between data and algorithms)
• Evaluation of unsupervised data mining (dilemma between novelty and validity)
• Evaluation measures
• (Automatic) data quality evaluation tools: What are the aspects one should check before starting to apply algorithms to given data?
• Issues around runtime evaluation (algorithm vs. implementation, dependency on hardware, algorithm parameters, dataset characteristics)
• Design guidelines for crowd-sourced evaluations
• Principled experimental workflows

The workshop will feature a mix of invited speakers, a number of accepted presentations with ample time for questions since those contributions will be less technical, and more philosophical in nature, and a panel discussion on the current state, and the areas that most urgently need improvement, as well as recommendations to achieve those improvements. An important objective of this workshop is a document synthesizing these discussions that we intend to publish at a prominent venue.

Submission

Papers should be submitted as PDF, using the Springer LNCS style, available at https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines. Submissions should be limited to ten pages and submitted via Easychair at https://easychair.org/conferences/?conf=edml20.

Papers will be reviewed by at least two members of the Program Committee on the basis of technical quality, relevance, significance, and clarity. Submitting a paper to the workshop means that if the paper is accepted at least one author should present the paper at the workshop. Accepted papers will be published after the workshop (e.g., with CEUR-WS or Springer).

Important dates

• Submission deadline: June 23, 2020
• Notification: July 14, 2020
• ECML PKDD early bird registration deadline: July 20, 2020 (but see https://ecmlpkdd2020.net/attending/registration/)
• Camera ready: July 21, 2020
• Conference dates: September 14-18, 2020
• Workshop date: September 14, 2020

Schedule

Note that the hosting conference ECML PKDD 2020, and thus also the EDML 2020 workshop, will be fully virtual. For more information see the conference information at https://ecmlpkdd2020.net/programme/conferenceformat/.

Accepted papers:

• Evgeniya Korneva and Hendrik Blockeel. Towards Better Evaluation of Multi-Target Regression Models
• Neetha Jambigi, Tirtha Chanda, Vishnu Unnikrishnan and Myra Spiliopoulou. Assessing the Difficulty of Labelling an Instance in Crowdworking
• Philipp Behnen, Rene Kessler, Felix Kruse, Jorge Marx Gómez, Jan Schoenmakers and Sergej Zerr. Experimental Evaluation of Scale, and Patterns of Systematic Inconsistencies in Google Trends Data
• Alexandru Mara, Jefrey Lijffijt and Tijl De Bie. Network Representation Learning for Link Prediction: Are we improving upon simple heuristics?
• Jonas Rieger, Carsten Jentsch and Jörg Rahnenführer. Assessing the Uncertainty of the Text Generating Process using Topic Models
• Lorenzo Perini, Connor Galvin and Vincent Vercruyssen. A Ranking Stability Measure for Quantifying the Robustness of Anomaly Detection Methods

Invited talks:

• Cecile Bothorel. Qualitative Evaluation in Graph Partitioning: How to guide a final user to choose a community detection method
• Ingo Thon. Machine Learning meets Software Development - A challenge for validation

Schedule and video presentations are available for registered participants over Whova. See https://ecmlpkdd2020.net/attending/participate/.

Proceedings

Special Issue

Organizers

• Eirini Ntoutsi
  Leibniz University Hannover & L3S Research Center, Germany
  ntoutsi@kbs.uni-hannover.de

  Eirini Ntoutsi is an Associate Professor in Intelligent Systems at the Faculty of Electrical Engineering and Computer Science at the Leibniz Universität Hannover (LUH) and member of the L3S research center. Prior to joining LUH, she was a postdoctoral researcher at the Ludwig-Maximilians-University (LMU) in Munich, Germany. She obtained her PhD from the University of Piraeus, Greece and she holds a diploma and a master in computer science from the Computer Engineering & Informatics Department (CEID), University of Patras, Greece. Her research can be summarized as learning over complex data (like high-dimensional, multi-view, with limited labels, ...) and data streams. She published more than 70 papers at international data mining and machine learning venues. She has organized several workshops, a Dagstuhl perspective workshop and served as publicity co-chair for ICDM 2017.

• Erich Schubert
  Technical University Dortmund
  erich.schubert@cs.tu-dortmund.de

  Erich Schubert is associate professor at the Technical University Dortmund, Germany in the Artificial Intelligence group. He joined TU Dortmund in 2018 after previous positions at Heidelberg University, Germany and Ludwig-Maximilians-Universität München (LMU Munich), Germany, where he obtained his PhD with Prof. Kriegel. His research interests include unsupervised learning, in particular clustering and outlier detection, along with index-based acceleration techniques for these approaches and evaluation methods. He has published over 38 peer reviewed papers at international conferences and in journals, and served as proceedings chair for the SISAP 2016 conference, and assisted with the GIR’17 workshop in Heidelberg.

• Arthur Zimek
  University of Southern Denmark
  zimek@imada.sdu.dk

  Arthur Zimek is Professor and Head of the Section "Data Science and Statistics" in the Department of Mathematics and Computer Science (IMADA) at University of Southern Denmark (SDU), in Odense, Denmark. He joined SDU in 2016 after previous positions at Ludwig-Maximilians-University Munich (LMU), Germany, Technical University Vienna, Austria, and University of Alberta, Edmonton, Canada. Arthur holds master-level degrees in bioinformatics, philosophy, and theology, involving studies at universities in Germany (TUM, HfPh, LMU Munich, and JGU Mainz) as well as Austria (LFU Innsbruck). His research interests include ensemble techniques for unsupervised learning, clustering, outlier detection, and high dimensional data, developing data mining methods as well as evaluation methodology. He published more than 90 papers at peer reviewed international conferences and in international journals. He co-organized several workshops and mini-symposia at SDM, KDD, and Shonan and served as workshop co-chair for SDM 2017.

• Albrecht Zimmermann
  University Caen Normandy, France
  albrecht.zimmermann@unicaen.fr

  Albrecht Zimmermann is associate professor at the University Caen Normandie, France in the CoDaG (Constraints, Data Mining, and Graphs) group. He joined the group in 2015 after previous stays at INSA Lyon, France, the Catholic University of Leuven, Belgium, and the Albert-Ludwigs University Freiburg, Germany. His research interests include pattern and pattern set mining, and their applications to bio- and chemoinformatics settings, result verification of unsupervised data mining methods, data generation, and sports analytics. He has co-organized four editions of the “Machine Learning and Data Mining for Sports Analytics” workshop @ ECML/PKDD (MLSA 2013, 2015, 2107, 2018), as well as three editions of the invitation-only “Spring Workshop on Mining and Learning” (SMiLe 2008, 2010, 2012). He served as registration chair of ICDM 2012, and as workshop and tutorial co-chair of ECML/PKDD 2016.

Program Committee

• Roberto Bayardo, Google
• Marcus Edel, Freie Universität Berlin
• Ricardo José Gabrielli Barreto Campello, University of Newcastle
• Fatih Gedikli, Hochschule Ruhr West
• Markus Goldstein, Ulm University of Applied Sciences
• Nathalie Japkowicz, American University
• Daniel Lemire, LICEF Research Center, Université du Québec
• Juergen Pfeffer, Technical University of Munich
• Miloš Radovanović, University of Novi Sad
• Maryam Tavakol, TU Dortmund

References

1. Basaran, Daniel, Eirini Ntoutsi, and Arthur Zimek. “Redundancies in Data and their Effect on the Evaluation of Recommendation Systems: A Case Study on the Amazon Reviews Datasets.” Proceedings of the 2017 SIAM International Conference on Data Mining. SIAM, 2017.
2. Kovács, Ferenc, Csaba Legány, and Attila Babos. "Cluster validity measurement techniques." 6th International symposium of hungarian researchers on computational intelligence. 2005.
3. Kriegel, Hans-Peter, Erich Schubert, and Arthur Zimek. "The (black) art of runtime evaluation: Are we comparing algorithms or implementations?" Knowledge and Information Systems 52.2 (2017): 341-378.
4. Nijssen, Siegfried, and Joost Kok. "Frequent subgraph miners: runtimes don't say everything." Proceedings of the Workshop on Mining and Learning with Graphs. 2006.
5. Zheng, Zijian, Ron Kohavi, and Llew Mason. "Real world performance of association rule algorithms." Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2001.
6. Muñoz, Villanova, Baatar, and Smith-Miles. "Instance spaces for machine learning classification." Machine Learning, 107.1 (2018), 109-147.