Extreme Algorithm Selection With Dyadic Feature Representation
TL;DR: This work assesses the applicability of state-of-the-art AS techniques to the XAS setting and proposes approaches leveraging a dyadic feature representation in which both problem instances and algorithms are described, finding the latter to improve significantly over the current state of the art in various metrics.
read more
Abstract: Algorithm selection (AS) deals with selecting an algorithm from a fixed set of candidate algorithms most suitable for a specific instance of an algorithmic problem, e.g., choosing solvers for SAT problems. Benchmark suites for AS usually comprise candidate sets consisting of at most tens of algorithms, whereas in combined algorithm selection and hyperparameter optimization problems the number of candidates becomes intractable, impeding to learn effective meta-models and thus requiring costly online performance evaluations. Therefore, here we propose the setting of extreme algorithm selection (XAS) where we consider fixed sets of thousands of candidate algorithms, facilitating meta learning. We assess the applicability of state-of-the-art AS techniques to the XAS setting and propose approaches leveraging a dyadic feature representation in which both problem instances and algorithms are described. We find the latter to improve significantly over the current state of the art in various metrics.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 3: Averaged results for the performance metrics Kendall’ tau (τ ), NDCG@k (N@3, N@5), and regret@k (R@1, R@3) for varying number of performance value pairs used for training. The best performing approach is highlighted in bold, the second best is underlined, and significant improvements of the best approach over others is denoted by •. 
Table 2: Overview of the data provided to the approaches and their applicability to the considered scenarios. 
Table 1: The table shows the types of classifiers used to derive the set A. Additionally, the number of numeric parameters (#num.P), categorical parameters (#cat.P), and instantiations (n) is shown.
Citations
A Survey of Methods for Automated Algorithm Configuration
Elias Arnold Schede,Jasmin Brandt,Alexander Tornede,Marcel Dominik Wever,Viktor Bengs,E. Hullermeier,Kevin Tierney +6 more
TL;DR: A review of existing AC literature within the lens of taxonomies, which outlines relevant design choices of configuration approaches, contrast methods and problem variants against each other, and provides a look at future research directions in the field of AC.
Hybrid Ranking and Regression for Algorithm Selection
Jonas Manuel Hanselle,Alexander Tornede,Marcel Dominik Wever,Eyke Hüllermeier +3 more
- 21 Sep 2020
TL;DR: A new approach to algorithm selection is developed that combines regression with ranking, also known as learning to rank, a problem that has recently been studied in the realm of preference learning and often performs better than pure regression and pure ranking methods.
14
Zero-Shot AutoML with Pretrained Models
Ekrem Ozturk,Fabio Ferreira,Hadi S. Jomaa,Lars Schmidt-Thieme,Josif Grabocka,Frank Hutter +5 more
- 16 Jun 2022
TL;DR: This work learns a zero-shot surrogate model, which, at test time, allows to select the right deep learning pipeline for a new dataset D given only trivial meta-features describing D, such as image resolution or the number of classes.
7
Algorithm Selection as Superset Learning: Constructing Algorithm Selectors from Imprecise Performance Data
Jonas Manuel Hanselle,Alexander Tornede,Marcel Dominik Wever,Eyke Hüllermeier +3 more
- 11 May 2021
TL;DR: In this paper, a simple regression method based on so-called superset learning is proposed, in which right-censored runtime data are explicitly incorporated in terms of interval-valued observations, offering an intuitive and efficient approach to handling censored data.
7
Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML
Lennart Purucker,Joeran Beel +1 more
TL;DR: The first version of Assembled-OpenML is described, a Python tool, which builds meta-datasets for ensembles using OpenML and implemented ensemble techniques expecting predictions instead of base models as input, to make the comparison of ensemble techniques computationally cheaper.
6
References
Using collaborative filtering to weave an information tapestry
TL;DR: Tapestry is intended to handle any incoming stream of electronic documents and serves both as a mail filter and repository; its components are the indexer, document store, annotation store, filterer, little box, remailer, appraiser and reader/browser.
4.7K
Sequential model-based optimization for general algorithm configuration
Frank Hutter,Holger H. Hoos,Kevin Leyton-Brown +2 more
- 17 Jan 2011
TL;DR: This paper extends the explicit regression models paradigm for the first time to general algorithm configuration problems, allowing many categorical parameters and optimization for sets of instances, and yields state-of-the-art performance.
Learning to rank: from pairwise approach to listwise approach
Zhe Cao,Tao Qin,Tie-Yan Liu,Ming-Feng Tsai,Hang Li +4 more
- 20 Jun 2007
TL;DR: It is proposed that learning to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning, and introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning.
2.4K
•Proceedings Article
Efficient and robust automated machine learning
Matthias Feurer,Aaron Klein,Katharina Eggensperger,Jost Tobias Springenberg,Manuel Blum,Frank Hutter +5 more
- 07 Dec 2015
TL;DR: This work introduces a robust new AutoML system based on scikit-learn, which improves on existing AutoML methods by automatically taking into account past performance on similar datasets, and by constructing ensembles from the models evaluated during the optimization.
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Chris Thornton,Frank Hutter,Holger H. Hoos,Kevin Leyton-Brown +3 more
- 11 Aug 2013
TL;DR: In this article, the problem of simultaneously selecting a learning algorithm and setting its hyperparameters is addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization, which can help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications.
Related Papers (5)
Maria-Florina Balcan,Travis Dick,Wesley Pegden +2 more
- 01 Aug 2020
Maria-Florina Balcan
- 01 Jan 2020