RT Journal Article
T1 TextFocus: Assessing the Faithfulness of Feature Attribution Methods Explanations in Natural Language Processing
A1 Mariotti, Ettore
A1 Arias-Duart, Anna
A1 Cafagna, Michele
A1 Gatt, Albert
A1 García-Gasulla, Darío
A1 Alonso Moral, José María
K1 Natural language processing
K1 Predictive models
K1 Measurement
K1 Explainable AI
K1 Data models
K1 Computational modeling
K1 Artificial intelligence
K1 Feature detection
K1 Modelos predictivos
K1 Modelos preditivos
K1 Modelos de datos
K1 Intelixencia Artificial
K1 Inteligencia Artificial
K1 Trustworthy AI
K1 Explanation faithfulness
AB Among the existing eXplainable AI (XAI) approaches, Feature Attribution methods are a popular option due to their interpretable nature. However, each method leads to a different solution, thus introducing uncertainty regarding their reliability and coherence with respect to the underlying model. This work introduces TextFocus, a metric for evaluating the faithfulness of Feature Attribution methods for Natural Language Processing (NLP) tasks involving classification. To address the absence of ground truth explanations for such methods, we introduce the concept of textual mosaics. A mosaic is composed of a combination of sentences belonging to different classes, which provides an implicit ground truth for attribution. The accuracy of explanations can be then evaluated by comparing feature attribution scores with the known class labels in the mosaic. The performance of six feature attribution methods is systematically compared on three sentence classification tasks by using TextFocus, with Integrated Gradients being the best overall method in terms of faithfulness and computational requirements. The proposed methodology fills a gap in NLP evaluation, by providing an objective way to assess Feature Attribution methods while finding their optimal parameters.
PB IEEE
SN 2169-3536
YR 2024
FD 2024-05-31
LK https://hdl.handle.net/10347/41177
UL https://hdl.handle.net/10347/41177
LA eng
NO E. Mariotti, A. Arias-Duart, M. Cafagna, A. Gatt, D. Garcia-Gasulla and J. M. Alonso-Moral. (2024). TextFocus: Assessing the Faithfulness of Feature Attribution Methods Explanations in Natural Language Processing. "IEEE Access", vol. 12, pp. 138870-138880
NO This work was supported in part by NL4XAI Project funded by European Union’s Horizon 2020 Research and Innovation Program under the Marie Skłodowska-Curie Grant under Agreement 860621; in part by MCIN/AEI/10.13039/501100011033 and 11ESF Investing in Your Future under Grant PID2021-123152OB-C21; in part by MCIN/AEI/10.13039/501100011033 and the 11European Union NextGenerationEU/PRTR under Grant TED2021-130295B-C33; in part by the Galician Ministry of Culture, Education, Professional Training, and University (co-funded by European Regional Development Fund, ERDF/FEDER Program) under Grant ED431G2019/04 and Grant ED431C2022/19; and in part by European Union–Horizon 2020 Program under the Scheme 11INFRAIA-01-2018-2019– Integrating Activities for Advanced Communities 11SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics (http://www.sobigdata.eu) under Grant 871042.
DS Minerva
RD 28 abr 2026