Assessing and Explaining Deep Learning Model Compression

Scientific Context

The success of deep neural networks is often constrained by their reliance on large amounts of labeled data, which are both costly and time-consuming to obtain. Prior work has demonstrated that models can be compressed by up to 84% without any loss in performance [4]. At the same time, self-supervised learning (SSL) has emerged as a promising alternative, enabling models to learn from unlabeled data. Moreover, SSL models offer the advantage of being adaptable to multiple downstream tasks thereby reducing the cost associated with training models for auxiliary tasks. In this context, we have proposed novel approaches for SSL model compression [15], particularly in the domains of speech recognition and emotion detection, achieving results that surpass the current state of the art.

However, the increasing complexity of deep neural networks raises significant challenges in terms of interpretability and trust, especially in sensitive domains such as medicine [5] and automotive systems. Explainable Artificial Intelligence (XAI) addresses these concerns by making model decisions more transparent and understandable [6]. The THM team has made notable contributions to XAI research, particularly through the RisKa project, which applies these methods to ECG analysis [5], and more recently through the SIGN method, designed to reduce bias in model explanations [7].

This thesis builds on and extends the work conducted by both teams. It is part of CESI-LINEACT’s Human-Machine Interaction research and aligns with the Future Industry and Future City domains. It also connects with THM-KITE’s research on explainable AI for signal and image processing, requiring strong collaboration and complementary expertise from both teams.Haut du formulaire

Bas du formulaire

Thesis abstract

Artificial intelligence (AI) has seen tremendous growth, becoming so omnipresent in our daily lives that intelligent applications are now integrated into our phones, vehicles, workplaces, and even our homes. These applications typically rely on large-scale deep neural network (DNN) models, as increasing model size often correlates with improved performance. However, the deployment of such models comes with significant computing and financial costs and contributes to a substantial carbon footprint. This not only challenges the inclusivity of AI [3] but also poses environmental concerns.

The success of DNNs is often limited by the need for vast amounts of labeled data, which can be both time-consuming and expensive to obtain. Self-supervised learning (SSL) emerges as a promising alternative, enabling models to learn from unlabeled data. SSL models have the advantage of being usable in multiple downstream tasks. Currently, few studies have focused on SSL compression [15]. Among the existing research on SSL compression, [9] applies knowledge distillation (KD) to the Wav2vec acoustic model, achieving a compression rate of 4.8 times. The authors report a word error rate (WER) that is 3.62 times higher than the original model, which is quite high for our applications. In [10], genetic algorithms are proposed for the structured pruning of Wav2vec2 XLSR53, and a slight increase in the word error rate of 0.21% (1.26% relative) is reported for a 40% pruning. The authors of [11] employ symmetric linear quantization to dynamically reduce the precision of weights and activations from FLOAT32 to INT8. They also explore quantization-aware training for the BERT language model. They find that post-training dynamic quantization slightly degrades performance, while quantization-aware training achieves performance comparable to the original model. To our knowledge, there is no research yet on the quantization of self-supervised speech models or on evaluating the effects of SSL pruning on auxiliary task performance.

Alternatively, XAI methods can significantly improve the transparency and understanding of speech processing models, whether for speech recognition, emotion detection, or speech-based language translation. Among these techniques are spectrogram heatmaps, which visualize the influential regions within the spectrogram; LIME (Local Interpretable Model-agnostic Explanations), which offers local explanations by simplifying a complex model around the prediction of an individual sample; and activation maximization, a method that visualizes the input features that most strongly activate a specific neuron within the network. In the existing literature [12] recommends using attention maps within complex transformer architectures to emphasize the audio signal segments most crucial for accurate speech recognition. The authors in [13] propose a method of Neuron Activation Profiles to explain model responses to certain groups of inputs. They investigate to what extent the model learns phonemes as an intermediate representation to predict graphemes, and show that phonemes are encoded in earlier layers than graphemes.

The PhD thesis addresses these challenges by investigating the combined potential of Green AI approaches and explainable AI (XAI) techniques. While Green AI provides methods to reduce computational costs and accelerate execution, XAI enables developers and users to better understand model decisions, identify biases, and debug unexpected behaviors. This work will systematically study this integration within the specific context of automatic speech recognition (ASR). In particular, it will explore the bidirectional relationship between model compression and explainability, with the goal of guiding both the selection of optimal compression methods and the tuning of associated strategies and parameters. Such systems will be more sustainable and suitable for deployment on edge devices, such as hearing aids. Furthermore, improved transparency in inference and on-device deployment will enhance trustworthiness and align with the requirements of the European AI Act.

Organisation

It is a joint French-German PhD thesis that will be registered in ENSAM (France) and in THM (Germany). Double PhD degree will be delivered conditioned on a successful PhD thesis.

Location:   18 months in France (CESI LINEACT – Campus de Nancy, France)

                  18 months in Germany (KITE – Friedberg, Germany)