Open AccessPosted Content
A Bandit-Based Algorithm for Fairness-Aware Hyperparameter Optimization.
TL;DR: It is shown that Fairband can efficiently navigate the fairness-accuracy trade-off through hyperparameter optimization, and consistently finds configurations attaining substantially improved fairness at a comparatively small decrease in predictive accuracy.
read more
Abstract: Considerable research effort has been guided towards algorithmic fairness but there is still no major breakthrough. In practice, an exhaustive search over all possible techniques and hyperparameters is needed to find optimal fairness-accuracy trade-offs. Hence, coupled with the lack of tools for ML practitioners, real-world adoption of bias reduction methods is still scarce. To address this, we present Fairband, a bandit-based fairness-aware hyperparameter optimization (HO) algorithm. Fairband is conceptually simple, resource-efficient, easy to implement, and agnostic to both the objective metrics, model types and the hyperparameter space being explored. Moreover, by introducing fairness notions into HO, we enable seamless and efficient integration of fairness objectives into real-world ML pipelines. We compare Fairband with popular HO methods on four real-world decision-making datasets. We show that Fairband can efficiently navigate the fairness-accuracy trade-off through hyperparameter optimization. Furthermore, without extra training cost, it consistently finds configurations attaining substantially improved fairness at a comparatively small decrease in predictive accuracy.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Figure 2: Average density of Pareto optimal models per FB-auto iteration (Adult dataset on left plot, AOF on right plot). Refer to Table 1 for information on the configurations at each iteration. 
Figure 3: Fairness and predictive accuracy of selected models by hyperparameter optimization algorithm (Adult dataset on left plot, AOF on right plot). 
Figure 1: Fairness-accuracy trade-off of thousands of models on the Adult dataset. In orange, the linear regression relationship between accuracy and fairness; in the red rectangle, the top 10% models with highest accuracy (the target region for a fairness-blind process); in light gray, the fairnessaccuracy Pareto frontier; marked with an A, the model with highest accuracy; marked with a B, a model with 0.8% lower accuracy and 44.8% higher fairness than A, arguably a better trade-off, and one that would not be found by traditional fairness-blind techniques. 
Table 4: Comparison of using Fairband (FB-auto) versus Hyperband (HB), on test results. 
Table 3: Validation and test results for all algorithms on all datasets. Statistical significance is tested against the baseline models (RS and HB) with a Kolmogorov-Smirnov test (Lilliefors, 1967). Comparison with RS is indicated by ♦ when p < 0.05 ( when p < 0.01), and comparison with HB is indicated by M when p < 0.05 (N when p < 0.01). 
Figure 5: Fairness and predictive accuracy as a function of budget in the AOF validation set.
Citations
Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service”
Kornel Lewicki,Michelle Seng Ah Lee,Jennifer Cobbe,Jatinder Singh +3 more
- 02 Feb 2023
TL;DR: In this paper , the authors review and systematise the AIaaS space by proposing a taxonomy of AI services based on the levels of autonomy afforded to the user, outlining how these services can lead to biases or be otherwise harmful in the context of end user applications.
Enforcing fairness using ensemble of diverse Pareto-optimal models
TL;DR: This study uses MOO methods to minimize the difference between groups, maximize the benefits for each group, and preserve performance in binary classification problems and finds models with higher fairness without sacrificing much accuracy.
6
Fairer and More Accurate Tabular Models Through NAS
TL;DR: This work proposes a novel approach that jointly optimizes architectural and training hyperparameters in a multi-objective constraint of both accuracy and fairness, and produces architectures that consistently Pareto dominate state-of-the-art bias mitigation methods either in fairness, accuracy or both.
•Posted Content
Promoting Fairness through Hyperparameter Optimization.
TL;DR: In this article, the authors proposed a fairness-aware hyperparameter optimization (HO) algorithm for real-world fraud detection, which enables practitioners to adapt pre-existing business operations to accommodate fairness objectives in a frictionless way.
1
References
Random Forests
Leo Breiman
- 01 Oct 2001
TL;DR: Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
•Book
Classification and regression trees
Leo Breiman
- 01 Jan 1983
TL;DR: The methodology used to construct tree structured rules is the focus of a monograph as mentioned in this paper, covering the use of trees as a data analysis method, and in a more mathematical framework, proving some of their fundamental properties.
22.7K
Classification and regression trees
TL;DR: This article gives an introduction to the subject of classification and regression trees by reviewing some widely available algorithms and comparing their capabilities, strengths, and weakness in two examples.
•Proceedings Article
LightGBM: a highly efficient gradient boosting decision tree
Guolin Ke,Qi Meng,Thomas Finley,Taifeng Wang,Wei Chen,Weidong Ma,Qiwei Ye,Tie-Yan Liu +7 more
- 04 Dec 2017
TL;DR: It is proved that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size, and is called LightGBM.
•Journal Article
Random search for hyper-parameter optimization
James Bergstra,Yoshua Bengio +1 more
TL;DR: This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid, and shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper- parameter optimization algorithms.