Joint BCAM-UPV/EHU Data Science and Artificial Intelligence seminar: Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results

LOCATION: UPV/EHU Donosti, Faculty of Computer Science, room 3.1 and Online

LInk to the session here

Abstract
Several recent works encourage the use of a Bayesian framework when assessing the performance and fairness metrics of a classification algorithm in a supervised setting. We have developed a framework called Uncertainty Matters (UM) that generalizes a Beta-Binomial approach to derive the posterior distribution of any criteria combination, allowing stable performance assessment in a bias-aware setting. In particular, we propose modeling the confusion matrix of each demographic group using a Multinomial distribution updated through a Bayesian procedure. We extend UM to be applicable under the popular K-fold cross-validation procedure. Experiments highlight the benefits of UM over classical evaluation frameworks regarding informativeness and stability.