Public defence: Quentin Mazouni

Background and Motivation

Decision-making systems have been shown to solve sequential tasks in various domains, such as manufacturing and automated driving. Consequently, they have been recently deployed in our society, for instance, think about automated vehicles.

These systems are prone to mistakes and hazard behaviours, whose consequences can be catastrophic. Indeed, given the infinite amount of situations, scenarios and environments that the decision models can interact with, their verification is impossible.

Testing is a lighter Validation and Verification methodology, which aims to find user-specified flaws within the models under test. Still, the main challenge is to drive the testing search towards fault-triggering test inputs, while maximizing additional testing objectives such as the identification of various dangerous behaviours.

Efficient automated testing frameworks of decision-making systems are pivotal for building trust in the models, and for fostering their safe deployments in our every-day life.

Objectives

The main objective of this research is to propose automated methods for testing decision-making systems that are tractable (efficient) and generally applicable. To accomplish this goal, four research questions (RQs) have been formulated:

• RQ1 How applicable is the state of the art in Reinforcement Learning (RL) policy testing?
• RQ2 How can we improve testing of RL policies for functional faults?
• RQ3 How to detect non-optimality in Artificial Intelligence (AI) planning systems?
• RQ4 How does stochasticity impact fault detection and test efficiency in RL policy testing?

Methods

This research proposed and employed methodologies that can be classified into three primary categories.

First, we reproduced and replicated a state-of-the-art policy testing framework. Not only this represents the first replicability study in policy testing, but it also lets us identify weaknesses and limitations in both the literature and the framework evaluated.

Second, we introduced a novel approach for testing these policies, that promotes the diversity of the faults through Quality Diversity (QD) optimization. In this context, we proposed the first formulation of policy testing as a QD task and examined various ways to characterize the behaviour of the model under test as well as different metrics for measuring diversity of both testing and the faults found.

Third, we developed another testing framework, based on metamorphic oracles, for detecting the lack of optimality in AI planning systems. In this work, we address test efficiency by selecting the most promising test cases with a mutation-based test case selection technique.

Contributions

The main contributions of this study include:

A Reproduction and Replication study of the state-of-the-art MDPFUZZ paper, whose primary outcomes are:

• Critical analysis of the evaluated approach.
• A two-step, reproduction and replication of the methodology, including a large-scale empirical evaluation of our peers’ work.
• Replicable and reusable artifacts, which will serve as an example for future research.
• Identification of shortcomings in the original evaluation, as well as limitations in the reproduced framework.
• Recommendation for future use of and works on MDPFUZZ.

A generally applicable testing framework for RL policies, that relies on Quality Diversity to promote fault diversity. In particular, we address the following settings:

• Deterministic test execution (the common assumption in the literature), where the policy is characterized by hand-specified behavioral description functions, hence requiring task-specific knowledge.
• Stochastic environments (i.e. real-world applications), for which each test is executed several times. We address the plurality of behaviors consequentially collected by introducing a domain-agnostic, testingfocused behavior space, aimed to cover the model’s reliability and action consistency

A metamorphic testing method for non-optimality detection in AI planning systems, which combines:

• Alleviating the oracle problem (i.e. the costs of the optimal plans are unknown) with metamorphic relations.
• Prioritizing the execution of the test cases with a mutation-based selection function, that scores each test case by its mutation score, defined as the proportion of flawed planners “killed” (i.e. detected) by the test case.
• Empirically evaluating the testing framework on both mutated AI planners and off-the-shelf planners.

Keywords

Software Testing, Decision-Making, Reinforcement Learning, AI Planning, Quality Diversity, Metamorphic Testing, Replicability.