Proceedings Article10.1145/3449726.3463299
Robust benchmarking for multi-objective optimization
Tome Eftimov,Peter Korošec +1 more
- 07 Jul 2021
- pp 9-10
2
TL;DR: In this paper, the authors extended the DSCTool with three approaches that are ensembles of quality indicators and one novel approach that compares the high-dimensional distributions of the approximation sets and reduces the users' preference in the selection of quality indicator.
read more
Abstract: The performance assessment of multi-objective optimization algorithms is a crucial task for investigating their behaviour. However, the selected quality indicators and statistical techniques used in comparison studies can have huge impact on the study results. A quality indicator transforms high-dimensional data (an approximation set) into one-dimensional data (a quality indicator), followed by a potential loss of high-dimensional information concerning the transformation. Comparison approaches typically involve a single quality indicator or an ensemble of quality indicators to address more quality criteria, which are predefined by the user. To provide more robust benchmarking for multi-objective optimization, we extended the DSCTool with three approaches that are ensembles of quality indicators and one novel approach that compare the high-dimensional distributions of the approximation sets and reduces the users' preference in the selection of quality indicators. The approaches are provided as web services for robust ranking and hypothesis testing, including a proper selection of an omnibus statistical test and post-hoc tests if needed.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Statistical Analyses for Single-objective Stochastic Optimization Algorithms
Tome Eftimov,Peter Korošec +1 more
- 14 Jul 2024
Statistical analyses for multi-objective stochastic optimization algorithms: GECCO 2022 tutorial
Tome Eftimov,Peter Korošec +1 more
- 09 Jul 2022
TL;DR: Tome Eftimov and Jožef Stefan as mentioned in this paper presented a tutorial on statistical analyses for multi-objective stochastic optimization algorithms at the GECCO 2022 tutorial workshop.
References
Benchmarking discrete optimization heuristics with IOHprofiler
TL;DR: This work compiles and assess a selection of 23 discrete optimization problems that subscribe to different types of fitness landscapes, and provides a new module for IOHprofiler which extents the fixed-target and fixed-budget results for the individual problems by ECDF results, which allows one to derive aggregated performance statistics for groups of problems.
67
Data-Driven Preference-Based Deep Statistical Ranking for Comparing Multi-objective Optimization Algorithms
Tome Eftimov,Peter Korošec,Barbara Koroušić Seljak +2 more
- 16 May 2018
TL;DR: This paper introduces a data-driven preference-based approach that is a combination of multiple criteria decision analysis with deep statistical rankings that ranks the algorithms for each benchmark problem using the preference (the influence) of each performance metric that is estimated using its entropy.
Nevergrad: black-box optimization platform
TL;DR: Nevergrad as mentioned in this paper is an open source platform for black-box optimization with a focus on optimization of black box optimization for optimization problems. And if you like Nevergrad, please support us by adding a star on GitHub (https://github.com/facebookresearch/nevergrad).
20
Comparing multi-objective optimization algorithms using an ensemble of quality indicators with deep statistical comparison approach
Tome Eftimov,Peter Korošec,Barbara Koroušić Seljak +2 more
- 01 Nov 2017
TL;DR: Experimental results performed using 3 multi-objective optimization algorithms on 16 test problems show that ensembles of quality indicators with transformed DSC rankings give more robust results than when the same ensembled are used with transformed rankings obtained by some standard ranking schemes.
Deep Statistical Comparison for Multi-Objective Stochastic Optimization Algorithms
Tome Eftimov,Peter Korošec +1 more
TL;DR: In this article, the authors proposed a ranking scheme that compares the distributions of high-dimensional data and showed that the proposed approach reduces potential information loss when statistical significance is not observed in highdimensional data.
10