An aggregated metrics framework for multicriteria model validation using rolling origin evaluation

Stanisław Halkiewicz and Mateusz Stachowicz

Abstract

This paper extends the rolling origin evaluation framework to model validation in multicriteria settings, where performance must be assessed across several scenarios or forecast targets. We propose three complementary metrics: the weighted sum of errors; the weighted aggregate performance metric; and the combined error and standard deviation metric. These metrics allow users to balance expected accuracy, fairness across scenarios and stability over repeated splits. A stress testing case study illustrates their practical value: the same gross domestic product growth series is forecast under baseline, adverse and prosperity scenarios, with supervisory-style weights reflecting regulatory priorities. The results show how each metric encodes a distinct evaluation philosophy and may recommend a different model depending on whether accuracy, balance or robustness is emphasized. We further introduce correlation-adjusted variants that penalize systemic errors across scenarios, ensuring that models vulnerable to structural shifts are not inadvertently selected. Together, these contributions provide a structured, quantitative framework for risk-aware model selection, supporting applications in finance, economics and other domains where scenario-based evaluation is essential.