Public defence: Quentin Mazouni
Quentin Mazouni will defend his thesis “Automated Testing of AI-Based Sequential Decision-Making” for the PhD in Engineering Science.
Quentin Mazouni will defend his thesis “Automated Testing of AI-Based Sequential Decision-Making” for the PhD in Engineering Science.
The candidate will defence the thesis 12.00 – 16.00.
The public defense will be streamed on Zoom (zoom.us).
Professor Magne Jørgensen, OsloMet
Associate Professor Way Kiat Bong, Department of Computer Science. OsloMet
Chief Research Scientist Arnaud Gotlieb, Simula Research Laboratory
Decision-making systems have been shown to solve sequential tasks in various domains, such as manufacturing and automated driving. Consequently, they have been recently deployed in our society, for instance, think about automated vehicles.
These systems are prone to mistakes and hazard behaviours, whose consequences can be catastrophic. Indeed, given the infinite amount of situations, scenarios and environments that the decision models can interact with, their verification is impossible.
Testing is a lighter Validation and Verification methodology, which aims to find user-specified flaws within the models under test. Still, the main challenge is to drive the testing search towards fault-triggering test inputs, while maximizing additional testing objectives such as the identification of various dangerous behaviours.
Efficient automated testing frameworks of decision-making systems are pivotal for building trust in the models, and for fostering their safe deployments in our every-day life.
The main objective of this research is to propose automated methods for testing decision-making systems that are tractable (efficient) and generally applicable. To accomplish this goal, four research questions (RQs) have been formulated:
This research proposed and employed methodologies that can be classified into three primary categories.
First, we reproduced and replicated a state-of-the-art policy testing framework. Not only this represents the first replicability study in policy testing, but it also lets us identify weaknesses and limitations in both the literature and the framework evaluated.
Second, we introduced a novel approach for testing these policies, that promotes the diversity of the faults through Quality Diversity (QD) optimization. In this context, we proposed the first formulation of policy testing as a QD task and examined various ways to characterize the behaviour of the model under test as well as different metrics for measuring diversity of both testing and the faults found.
Third, we developed another testing framework, based on metamorphic oracles, for detecting the lack of optimality in AI planning systems. In this work, we address test efficiency by selecting the most promising test cases with a mutation-based test case selection technique.
The main contributions of this study include:
A Reproduction and Replication study of the state-of-the-art MDPFUZZ paper, whose primary outcomes are:
A generally applicable testing framework for RL policies, that relies on Quality Diversity to promote fault diversity. In particular, we address the following settings:
A metamorphic testing method for non-optimality detection in AI planning systems, which combines:
Software Testing, Decision-Making, Reinforcement Learning, AI Planning, Quality Diversity, Metamorphic Testing, Replicability.
