Content deleted Content added
cleaned the external links (one was redundant, another one dead) |
Citation bot (talk | contribs) Alter: title, pages. Add: bibcode, article-number. Removed URL that duplicated identifier. Removed parameters. Formatted dashes. Some additions/deletions were parameter name changes. | Use this bot. Report bugs. | Suggested by 16dvnk | Category:Artificial intelligence | #UCB_Category 48/198 |
||
| (40 intermediate revisions by 25 users not shown) | |||
Line 3:
{{artificial intelligence}}
Within [[artificial intelligence]] (AI), '''
XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason.<ref>{{Cite journal|last=Alizadeh|first=Fatemeh|date=2021|title=I Don't Know, Is AI Also Used in Airbags?: An Empirical Study of Folk Concepts and People's Expectations of Current and Future Artificial Intelligence|url=https://www.researchgate.net/publication/352638184|journal=Icom|volume=20 |issue=1 |pages=3–17 |doi=10.1515/icom-2021-0009|s2cid=233328352}}</ref> XAI may be an implementation of the social [[right to explanation]].<ref name=":0">{{Cite journal|last1=Edwards|first1=Lilian|last2=Veale|first2=Michael|date=2017|title=Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For|journal=Duke Law and Technology Review|volume=16|pages=18|ssrn=2972855}}</ref> Even if there is no such legal right or regulatory requirement, XAI can improve the [[user experience]] of a product or service by helping end users trust that the AI is making good decisions.<ref>{{Cite web |last=Do Couto |first=Mark |date=February 22, 2024 |title=Entering the Age of Explainable AI |url=https://tdwi.org/Articles/2024/02/22/ADV-ALL-Entering-the-Age-of-Explainable-AI.aspx |access-date=2024-09-11 |website=TDWI}}</ref> XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on.<ref name=":3">{{Cite journal|last1=Gunning|first1=D.|last2=Stefik|first2=M.|last3=Choi|first3=J.|last4=Miller|first4=T.|last5=Stumpf|first5=S.|last6=Yang|first6=G.-Z.|date=2019-12-18|title=XAI-Explainable artificial intelligence|url=https://openaccess.city.ac.uk/id/eprint/23405/|journal=Science Robotics|language=en|volume=4|issue=37|pages=eaay7120|doi=10.1126/scirobotics.aay7120|pmid=33137719|issn=2470-9476|doi-access=free}}</ref> This makes it possible to confirm existing knowledge, challenge existing knowledge, and generate new assumptions.<ref>{{Cite journal|last1=Rieg|first1=Thilo|last2=Frick|first2=Janek|last3=Baumgartl|first3=Hermann|last4=Buettner|first4=Ricardo|date=2020-12-17|title=Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms|journal=PLOS ONE|language=en|volume=15|issue=12|pages=e0243615|doi=10.1371/journal.pone.0243615|issn=1932-6203|pmc=7746264|pmid=33332440|bibcode=2020PLoSO..1543615R|doi-access=free}}</ref>
==Background==
[[Machine learning]] (ML) algorithms used in AI can be categorized as [[White-box testing|white-box]] or [[Black box|black-box]].<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title= Classification of Explainable Artificial Intelligence Methods through Their Output Formats |journal=Machine Learning and Knowledge Extraction|year=2021|volume=3|issue=3|pages=615–661|doi=10.3390/make3030032|doi-access=free }}</ref> White-box models provide results that are understandable to experts in the ___domain. Black-box models, on the other hand, are extremely hard to explain and may not be understood even by ___domain experts.<ref>{{Cite journal|last=Loyola-González|first=O.|date=2019|title=Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View|journal=IEEE Access|volume=7|pages=154096–154113|doi=10.1109/ACCESS.2019.2949286|bibcode=2019IEEEA...7o4096L |issn=2169-3536|doi-access=free}}</ref> XAI algorithms follow the three principles of transparency, interpretability, and explainability.
* A model is transparent "if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer."<ref name=":4">{{Cite journal|last1=Roscher|first1=R.|last2=Bohn|first2=B.|last3=Duarte|first3=M. F.|last4=Garcke|first4=J.|date=2020|title=Explainable Machine Learning for Scientific Insights and Discoveries|journal=IEEE Access|volume=8|pages=42200–42216|doi=10.1109/ACCESS.2020.2976199|arxiv=1905.08883 |bibcode=2020IEEEA...842200R |issn=2169-3536|doi-access=free}}</ref>
Sometimes it is also possible to achieve a high-accuracy result with white-box ML algorithms. These algorithms have an interpretable structure that can be used to explain predictions.<ref name=":6">{{Cite journal|last=Rudin|first=Cynthia|date=2019|title=Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead|journal=Nature Machine Intelligence|language=en|volume=1|issue=5|pages=206–215|doi=10.1038/s42256-019-0048-x|pmid=35603010 |pmc=9122117 |arxiv=1811.10154|issn=2522-5839|doi-access=free}}</ref> Concept Bottleneck Models, which use concept-level abstractions to explain model reasoning, are examples of this and can be applied in both image<ref name="Koh Nguyen Tang Mussmann Pierson Kim Liang 2020">{{Cite conference|last1=Koh|first1=P. W.|last2=Nguyen|first2=T.|last3=Tang|first3=Y. S.|last4=Mussmann|first4=S.|last5=Pierson|first5=E.|last6=Kim|first6=B.|last7=Liang|first7=P.|date=November 2020|title=Concept bottleneck models|book-title=International Conference on Machine Learning|pages=5338–5348|publisher=PMLR}}</ref> and text<ref name="Ludan Lyu Yang Dugan Yatskar Callison-Burch 2023">{{Cite arXiv|last1=Ludan|first1=J. M.|last2=Lyu|first2=Q.|last3=Yang|first3=Y.|last4=Dugan|first4=L.|last5=Yatskar|first5=M.|last6=Callison-Burch|first6=C.|date=2023|title=Interpretable-by-Design Text Classification with Iteratively Generated Concept Bottleneck|class=cs.CL |eprint=2310.19660}}</ref> prediction tasks. This is especially important in domains like medicine, defense, finance, and law, where it is crucial to understand decisions and build trust in the algorithms.<ref name=":3" /> Many researchers argue that, at least for supervised machine learning, the way forward is symbolic regression, where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.<ref name="Wenninger Kaymakci Wiethe 2022 p=118300">{{cite journal | last1=Wenninger | first1=Simon | last2=Kaymakci | first2=Can | last3=Wiethe | first3=Christian | title=Explainable long-term building energy consumption prediction using QLattice | journal=Applied Energy | publisher=Elsevier BV | volume=308 | year=2022 | issn=0306-2619 | doi=10.1016/j.apenergy.2021.118300 | page=118300| bibcode=2022ApEn..30818300W | s2cid=245428233 }}</ref><ref name="Christiansen Wilstrup Hedley 2022 p.">{{cite journal | last1=Christiansen | first1=Michael | last2=Wilstrup | first2=Casper | last3=Hedley | first3=Paula L. | title=Explainable "white-box" machine learning is the way forward in preeclampsia screening | journal=American Journal of Obstetrics and Gynecology | publisher=Elsevier BV | year=2022 | volume=227 | issue=5 | issn=0002-9378 | doi=10.1016/j.ajog.2022.06.057 | page=791| pmid=35779588 | s2cid=250160871 }}</ref><ref name="Wilstup Cave p.">{{citation | last1=Wilstup | first1=Casper | last2=Cave | first2=Chris | title=Combining symbolic regression with the Cox proportional hazards model improves prediction of heart failure deaths | publisher=Cold Spring Harbor Laboratory | date=2021-01-15 | doi=10.1101/2021.01.15.21249874 | page=| s2cid=231609904 }}</ref>▼
* Interpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.<ref name="Interpretable machine learning: def">{{cite journal|last1=Murdoch|first1=W. James|last2=Singh|first2=Chandan|last3=Kumbier|first3=Karl|last4=Abbasi-Asl|first4=Reza|last5=Yu|first5=Bin|date=2019-01-14|title=Interpretable machine learning: definitions, methods, and applications|journal=Proceedings of the National Academy of Sciences of the United States of America|volume=116|issue=44|pages=22071–22080|arxiv=1901.04592|doi=10.1073/pnas.1900654116|pmid=31619572|pmc=6825274|bibcode= |doi-access=free}}</ref><ref name="Lipton 31–57">{{Cite journal|last=Lipton|first=Zachary C.|date=June 2018|title=The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.|journal=Queue|language=en|volume=16|issue=3|pages=31–57|doi=10.1145/3236386.3241340|issn=1542-7730|doi-access=free|arxiv=1606.03490}}</ref><ref>{{Cite web|date=2019-10-22|title=Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI|url=https://deepai.org/publication/explainable-artificial-intelligence-xai-concepts-taxonomies-opportunities-and-challenges-toward-responsible-ai|access-date=2021-01-13|website=DeepAI}}</ref>
* Explainability is a concept that is recognized as important, but a consensus definition is not yet available;<ref name=":4" /> one possibility is "the collection of features of the interpretable ___domain that have contributed, for a given example, to producing a decision (e.g., classification or regression)".<ref>{{Cite journal|date=2018-02-01|title=Methods for interpreting and understanding deep neural networks|journal=Digital Signal Processing|language=en|volume=73|pages=1–15|doi=10.1016/j.dsp.2017.10.011|issn=1051-2004|doi-access=free|last1=Montavon|first1=Grégoire|last2=Samek|first2=Wojciech|last3=Müller|first3=Klaus-Robert|arxiv=1706.07979 |bibcode=2018DSP....73....1M |author-link3=Klaus-Robert Müller}}</ref>
In summary, Interpretability refers to the user's ability to understand model outputs, while Model Transparency includes Simulatability (reproducibility of predictions), Decomposability (intuitive explanations for parameters), and Algorithmic Transparency (explaining how algorithms work). Model Functionality focuses on textual descriptions, visualization, and local explanations, which clarify specific outputs or instances rather than entire models. All these concepts aim to enhance the comprehensibility and usability of AI systems.<ref name="NCB23">{{Cite web|author = Notovich, Aviv., Chalutz-Ben Gal, Hila, & Ben-Gal, Irad (2023). |title = Explainable Artificial Intelligence (XAI): Motivation, Terminology, and Taxonomy.|url = https://www.iradbengal.sites.tau.ac.il/_files/ugd/901879_fcdc9f3f44dc41968ba476a66f60f078.pdf|publisher = In Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook (pp. 971-985). Cham: Springer International Publishing.}}</ref>
If algorithms fulfill these principles, they provide a basis for justifying decisions, tracking them and thereby verifying them, improving the algorithms, and exploring new facts.<ref>{{Cite journal|last1=Adadi|first1=A.|last2=Berrada|first2=M.|date=2018|title=Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)|journal=IEEE Access|volume=6|pages=52138–52160|doi=10.1109/ACCESS.2018.2870052|bibcode=2018IEEEA...652138A |issn=2169-3536|doi-access=free}}</ref>
▲Sometimes it is also possible to achieve a high-accuracy result with white-box ML algorithms. These algorithms have an interpretable structure that can be used to explain predictions.<ref name=":6">{{Cite journal|last=Rudin|first=Cynthia|date=2019|title=Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead|journal=Nature Machine Intelligence|language=en|volume=1|issue=5|pages=206–215|doi=10.1038/s42256-019-0048-x|pmid=35603010 |pmc=9122117 |arxiv=1811.10154|issn=2522-5839|doi-access=free}}</ref> Concept Bottleneck Models, which use concept-level abstractions to explain model reasoning, are examples of this and can be applied in both image<ref name="Koh Nguyen Tang Mussmann Pierson Kim Liang 2020">{{Cite conference|last1=Koh|first1=P. W.|last2=Nguyen|first2=T.|last3=Tang|first3=Y. S.|last4=Mussmann|first4=S.|last5=Pierson|first5=E.|last6=Kim|first6=B.|last7=Liang|first7=P.|date=November 2020|title=Concept bottleneck models|book-title=International Conference on Machine Learning|pages=5338–5348|publisher=PMLR}}</ref> and text<ref name="Ludan Lyu Yang Dugan Yatskar Callison-Burch 2023">{{Cite arXiv|last1=Ludan|first1=J. M.|last2=Lyu|first2=Q.|last3=Yang|first3=Y.|last4=Dugan|first4=L.|last5=Yatskar|first5=M.|last6=Callison-Burch|first6=C.|date=2023|title=Interpretable-by-Design Text Classification with Iteratively Generated Concept Bottleneck|class=cs.CL |eprint=2310.19660}}</ref> prediction tasks. This is especially important in domains like [[medicine]], [[Defense industry|defense]], [[finance]], and [[law]], where it is crucial to understand decisions and build trust in the algorithms.<ref name=":3" /> Many researchers argue that, at least for [[supervised machine learning]], the way forward is [[symbolic regression]], where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.<ref name="Wenninger Kaymakci Wiethe 2022 p=118300">{{cite journal | last1=Wenninger | first1=Simon | last2=Kaymakci | first2=Can | last3=Wiethe | first3=Christian | title=Explainable long-term building energy consumption prediction using QLattice | journal=Applied Energy | publisher=Elsevier BV | volume=308 | year=2022 | issn=0306-2619 | doi=10.1016/j.apenergy.2021.118300 |
AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command "maximize the accuracy of [[sentiment analysis|assessing how positive]] film reviews are in the test dataset." The AI may learn useful general rules from the test set, such as "reviews containing the word "horrible" are likely to be negative." However, it may also learn inappropriate rules, such as "reviews containing '[[Daniel Day-Lewis]]' are usually positive"; such rules may be undesirable if they are likely to fail to generalize outside the training set, or if people consider the rule to be "cheating" or "unfair." A human can audit rules in an XAI to get an idea of how likely the system is to generalize to future real-world data outside the test set.<ref name="science">{{cite journal|date=5 July 2017|title=How AI detectives are cracking open the black box of deep learning|url=https://www.science.org/content/article/how-ai-detectives-are-cracking-open-black-box-deep-learning|journal=Science|language=en|access-date=30 January 2018}}.</ref>
Line 41 ⟶ 49:
* ''Feature importance'' estimates how important a feature is for the model. It is usually done using ''permutation importance'', which measures the performance decrease when it the feature value randomly shuffled across all samples.
* ''LIME'' approximates locally a model's outputs with a simpler, interpretable model.<ref>{{Cite web |last=Rothman |first=Denis |date=2020-10-07 |title=Exploring LIME Explanations and the Mathematics Behind It |url=https://www.codemotion.com/magazine/ai-ml/lime-explainable-ai/ |access-date=2024-07-10 |website=Codemotion Magazine |language=en-US}}</ref>
* ''[[Multitask learning
For images, [[Saliency map|saliency maps]] highlight the parts of an image that most influenced the result.<ref>{{Cite web |last=Sharma |first=Abhishek |date=2018-07-11 |title=What Are Saliency Maps In Deep Learning? |url=https://analyticsindiamag.com/what-are-saliency-maps-in-deep-learning/ |access-date=2024-07-10 |website=Analytics India Magazine |language=en-US}}</ref>
Systems that are expert or knowledge based are software systems that are made by experts. This system consists of a knowledge based encoding for the ___domain knowledge. This system is usually modeled as production rules, and someone uses this knowledge base which the user can question the system for knowledge. In expert systems, the language and explanations are understood with an explanation for the reasoning or a problem solving activity.<ref name="auto"/>
However, these techniques are not very suitable for [[Language model|language models]] like [[Generative pre-trained transformer|generative pretrained transformers]]. Since these models generate language, they can provide an explanation, but which may not be reliable. Other techniques include attention analysis (examining how the model focuses on different parts of the input), probing methods (testing what information is captured in the model's representations), causal tracing (tracing the flow of information through the model) and circuit discovery (identifying specific subnetworks responsible for certain behaviors). Explainability research in this area overlaps significantly with interpretability and [[AI alignment|alignment]] research.<ref>{{cite arXiv |last1=Luo |first1=Haoyan |title=From Understanding to Utilization: A Survey on Explainability for Large Language Models |date=2024-02-21 |eprint=2401.12874 |last2=Specia |first2=Lucia|class=cs.CL }}</ref>▼
▲However, these techniques are not very suitable for [[Language model|language models]] like [[Generative pre-trained transformer|generative pretrained transformers]]. Since these models generate language, they can provide an explanation, but which may not be reliable. Other techniques include [[Attention (machine learning)|attention]] analysis (examining how the model focuses on different parts of the input), probing methods (testing what information is captured in the model's representations), causal tracing (tracing the flow of information through the model) and circuit discovery (identifying specific subnetworks responsible for certain behaviors). Explainability research in this area overlaps significantly with interpretability and [[AI alignment|alignment]] research.<ref>{{cite arXiv |last1=Luo |first1=Haoyan |title=From Understanding to Utilization: A Survey on Explainability for Large Language Models |date=2024-02-21 |eprint=2401.12874 |last2=Specia |first2=Lucia|class=cs.CL }}</ref>
=== Interpretability ===
[[File:Grokking modular addition.jpg|thumb|upright=1.2|[[Grokking (machine learning)|Grokking]] is an example of phenomenon studied in interpretability. It involves a model that initially memorizes all the answers ([[overfitting]]), but later adopts an algorithm that generalizes to unseen data.<ref>{{Cite web |last=Ananthaswamy |first=Anil |date=2024-04-12 |title=How Do Machines 'Grok' Data? |url=https://www.quantamagazine.org/how-do-machines-grok-data-20240412/ |access-date=2025-01-21 |website=Quanta Magazine |language=en}}</ref>]]
Scholars sometimes use the term "[[mechanistic interpretability]]" to refer to the process of [[Reverse engineering|reverse-engineering]] [[artificial neural networks]] to understand their internal decision-making mechanisms and components, similar to how one might analyze a complex machine or computer program.<ref>{{Cite web |last=Olah |first=Chris |date=June 27, 2022 |title=Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases |url=https://www.transformer-circuits.pub/2022/mech-interp-essay |access-date=2024-07-10 |website=www.transformer-circuits.pub}}</ref>
Interpretability research often focuses on generative pretrained transformers. It is particularly relevant for [[AI safety]] and [[AI alignment|alignment
Studying the interpretability of the most advanced [[Foundation model|foundation models]] often involves searching for an automated way to identify "features" in generative pretrained transformers. In a [[Neural network (machine learning)|neural network]], a feature is a pattern of neuron activations that corresponds to a concept. A compute-intensive technique called "[[dictionary learning]]" makes it possible to identify features to some degree. Enhancing the ability to identify and edit features is expected to significantly improve the [[AI safety|safety]] of [[Frontier model|frontier AI models]].<ref>{{Cite web |last=Ropek |first=Lucas |date=2024-05-21 |title=New Anthropic Research Sheds Light on AI's 'Black Box' |url=https://gizmodo.com/new-anthropic-research-sheds-light-on-ais-black-box-1851491333 |access-date=2024-05-23 |website=Gizmodo |language=en}}</ref><ref>{{Cite magazine |last=Perrigo |first=Billy |date=2024-05-21 |title=Artificial Intelligence Is a 'Black Box.' Maybe Not For Long |url=https://time.com/6980210/anthropic-interpretability-ai-safety-research/ |access-date=2024-05-24 |magazine=Time |language=en}}</ref>
Line 69 ⟶ 80:
[[Marvin Minsky]] et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.<ref>Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.</ref>
Explainable AI has been recently a new topic researched amongst the context of modern deep learning. Modern complex AI techniques, such as [[deep learning]], are naturally opaque.<ref>{{cite magazine|last1=Mukherjee|first1=Siddhartha|title=A.I. Versus M.D.|url=https://www.newyorker.com/magazine/2017/04/03/ai-versus-md|access-date=30 January 2018|magazine=The New Yorker|date=27 March 2017}}</ref> To address this issue, methods have been developed to make new models more explainable and interpretable.<ref>{{Cite journal|date=2020-07-08|title=Interpretable neural networks based on continuous-valued logic and multicriteria decision operators|journal=Knowledge-Based Systems|language=en|volume=199|
There has been work on making glass-box models which are more transparent to inspection.<ref name=":6"/><ref>{{cite journal |last1=Singh |first1=Chandan |last2=Nasseri |first2=Keyan |last3=Tan |first3=Yan Shuo |last4=Tang |first4=Tiffany |last5=Yu |first5=Bin |title=imodels: a python package for fitting interpretable models |journal=Journal of Open Source Software |date=4 May 2021 |volume=6 |issue=61 |pages=3192 |doi=10.21105/joss.03192 |bibcode=2021JOSS....6.3192S |s2cid=235529515
Some techniques allow visualisations of the inputs to which individual [[Neuron (software)|software neurons]] respond to most strongly. Several groups found that neurons can be aggregated into circuits that perform human-comprehensible functions, some of which reliably arise across different networks trained independently.<ref name="Circuits">{{cite journal |last1=Olah |first1=Chris |last2=Cammarata |first2=Nick |last3=Schubert |first3=Ludwig |last4=Goh |first4=Gabriel |last5=Petrov |first5=Michael |last6=Carter |first6=Shan |title=Zoom In: An Introduction to Circuits |journal=Distill |date=10 March 2020 |volume=5 |issue=3 |pages=e00024.001 |doi=10.23915/distill.00024.001 |url=https://distill.pub/2020/circuits/zoom-in/ |language=en |issn=2476-0757|doi-access=free }}</ref><ref>{{cite journal |last1=Li |first1=Yixuan |last2=Yosinski |first2=Jason |last3=Clune |first3=Jeff |last4=Lipson |first4=Hod |last5=Hopcroft |first5=John |title=Convergent Learning: Do different neural networks learn the same representations? |journal=Feature Extraction: Modern Questions and Challenges |date=8 December 2015 |pages=196–212 |url=http://proceedings.mlr.press/v44/li15convergent.html |publisher=PMLR |language=en}}</ref>
There are various techniques to extract compressed representations of the features of given inputs, which can then be analysed by standard [[Cluster analysis|clustering techniques]]. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable.<ref>{{cite book |last1=Hendricks |first1=Lisa Anne |last2=Akata |first2=Zeynep |last3=Rohrbach |first3=Marcus |last4=Donahue |first4=Jeff |last5=Schiele |first5=Bernt |last6=Darrell |first6=Trevor |title=Computer Vision – ECCV 2016 |chapter=Generating Visual Explanations |series=Lecture Notes in Computer Science |date=2016 |volume=9908 |pages=3–19 |doi=10.1007/978-3-319-46493-0_1 |chapter-url=https://link.springer.com/chapter/10.1007/978-3-319-46493-0_1 |publisher=Springer International Publishing |language=en|arxiv=1603.08507 |isbn=978-3-319-46492-3 |s2cid=12030503 }}</ref> Model behaviour can also be explained with reference to training data—for example, by evaluating which training inputs influenced a given behaviour the most
The use of explainable artificial intelligence (XAI) in pain research, specifically in understanding the role of electrodermal activity for [[Automated Pain Recognition|automated pain recognition]]: hand-crafted features and deep learning models in pain recognition, highlighting the insights that simple hand-crafted features can yield comparative performances to deep learning models and that both traditional feature engineering and deep feature learning approaches rely on simple characteristics of the input time-series data.<ref>{{Cite journal |last1=Gouverneur |first1=Philip |last2=Li |first2=Frédéric |last3=Shirahama |first3=Kimiaki |last4=Luebke |first4=Luisa |last5=Adamczyk |first5=Wacław M. |last6=Szikszay |first6=Tibor M. |last7=Luedtke |first7=Kerstin |last8=Grzegorzek |first8=Marcin |date=2023-02-09 |title=Explainable Artificial Intelligence (XAI) in Pain Research: Understanding the Role of Electrodermal Activity for Automated Pain Recognition |journal=Sensors |language=en |volume=23 |issue=4 |pages=1959 |doi=10.3390/s23041959 |issn=1424-8220 |pmc=9960387 |pmid=36850556 |bibcode=2023Senso..23.1959G |doi-access=free }}</ref>
Line 81 ⟶ 92:
== Regulation ==
As regulators, official bodies, and general users come to depend on AI-based dynamic systems, clearer accountability will be required for [[automated decision-making]] processes to ensure trust and transparency. The first global conference exclusively dedicated to this emerging discipline was the 2017 [[International Joint Conference on Artificial Intelligence]]: Workshop on Explainable Artificial Intelligence (XAI).<ref>{{cite web|title=IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)|url=http://www.intelligentrobots.org/files/IJCAI2017/IJCAI-17_XAI_WS_Proceedings.pdf|website=Earthlink|publisher=IJCAI|access-date=17 July 2017|archive-date=4 April 2019|archive-url=https://web.archive.org/web/20190404131609/http://www.intelligentrobots.org/files/IJCAI2017/IJCAI-17_XAI_WS_Proceedings.pdf|url-status=dead}}</ref> It has evolved over the years, with various workshops organised and co-located to many other international conferences, and it has now a dedicated global event, "The world conference on eXplainable Artificial Intelligence", with its own proceedings.<ref name="XAI-2023">{{cite book |author=<!--Not stated--> |date= 2023| title= Explainable Artificial Intelligence, First World Conference, xAI 2023, Lisbon, Portugal, July 26–28, 2023, Proceedings, Parts I/II/III |series= Communications in Computer and Information Science|volume= 1903|url= https://link.springer.com/book/10.1007/978-3-031-44070-0 |publisher=springer |doi= 10.1007/978-3-031-44070-0|isbn=978-3-031-44070-0}}</ref><ref name="XAI-2024">{{cite book |author=<!--Not stated-->|date= 2024| title= Explainable Artificial Intelligence, Second World Conference, xAI 2024, Valletta, Malta, July 17–19, 2024, Proceedings, Part I/II/III/IV |series= Communications in Computer and Information Science|volume= 2153|url=https://link.springer.com/book/10.1007/978-3-031-63787-2 |publisher=springer |doi= 10.1007/978-3-031-63787-2|isbn=978-3-031-63787-2}}</ref>
The European Union introduced a [[right to explanation]] in the [[General Data Protection Regulation
▲The European Union introduced a [[right to explanation]] in the [[General Data Protection Regulation|General Data Protection Right (GDPR)]] to address potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions.<ref>{{cite news |last1=Kahn |first1=Jeremy |title=Artificial Intelligence Has Some Explaining to Do |url=https://www.bloomberg.com/news/articles/2018-12-12/artificial-intelligence-has-some-explaining-to-do |access-date=17 December 2018 |work=[[Bloomberg Businessweek]] |date=12 December 2018}}</ref> In France the [[Loi pour une République numérique]] (Digital Republic Act) grants subjects the right to request and receive information pertaining to the implementation of algorithms that process data about them.
== Limitations ==
Line 92 ⟶ 102:
By making an AI system more explainable, we also reveal more of its inner workings. For example, the explainability method of feature importance identifies features or variables that are most important in determining the model's output, while the influential samples method identifies the training samples that are most influential in determining the output, given a particular input.<ref name="Explainable Machine Learning in Deployment">{{cite book | last1=Bhatt | first1=Umang | last2=Xiang | first2=Alice | last3=Sharma | first3=Shubham | last4=Weller | first4=Adrian | last5=Taly | first5=Ankur | last6=Jia | first6=Yunhan | last7=Ghosh | first7=Joydeep | last8=Puri | first8=Richir | last9=M.F. Moura | first9=José | last10=Eckersley | first10=Peter | title=Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency | chapter=Explainable Machine Learning in Deployment | date=2022 | pages=648–657 | doi=10.1145/3351095.3375624 | isbn=9781450369367 | s2cid=202572724 | chapter-url=https://dl.acm.org/doi/pdf/10.1145/3351095.3375624 }}</ref> Adversarial parties could take advantage of this knowledge.
For example, competitor firms could replicate aspects of the original AI system in their own product, thus reducing competitive advantage.<ref name="How the machine 'thinks'">{{cite journal |
=== Adaptive integration and explanation ===
Many approaches that it uses provides explanation in general, it doesn't take account for the diverse backgrounds and knowledge level of the users. This leads to challenges with accurate comprehension for all users. Expert users can find the explanations lacking in depth, and are oversimplified, while a beginner user may struggle understanding the explanations as they are complex. This limitation downplays the ability of the XAI techniques to appeal to their users with different levels of knowledge, which can impact the trust from users and who uses it. The quality of explanations can be different amongst their users as they all have different expertise levels, including different situation and conditions.<ref>{{Cite journal |last1=Yang |first1=Wenli |last2=Wei |first2=Yuchen |last3=Wei |first3=Hanyu |last4=Chen |first4=Yanyu |last5=Huang |first5=Guan |last6=Li |first6=Xiang |last7=Li |first7=Renjie |last8=Yao |first8=Naimeng |last9=Wang |first9=Xinyi |last10=Gu |first10=Xiaotong |last11=Amin |first11=Muhammad Bilal |last12=Kang |first12=Byeong |date=2023-08-10 |title=Survey on Explainable AI: From Approaches, Limitations and Applications Aspects |journal=Human-Centric Intelligent Systems |language=en |volume=3 |issue=3 |pages=161–188 |doi=10.1007/s44230-023-00038-y |issn=2667-1336|doi-access=free }}</ref>
=== Technical complexity ===
Line 100 ⟶ 113:
=== Understanding versus trust ===
The goal of explainability to end users of AI systems is to increase trust in the systems, even “address concerns about lack of ‘fairness’ and discriminatory effects”.<ref name="How the machine 'thinks'" /> However, even with a good understanding of an AI system, end users may not necessarily trust the system.<ref>{{cite journal |last1=Hu |first1=Tongxi |last2=Zhang |first2=Xuesong |last3=Bohrer |first3=Gil |last4=Liu |first4=Yanlan |last5=Zhou |first5=Yuyu |last6=Martin |first6=Jay |last7=LI |first7=Yang |last8=Zhao |first8=Kaiguang |title=Crop yield prediction via explainable AI and interpretable machine learning: Dangers of black box models for evaluating climate change impacts on crop yield|journal=Agricultural and Forest Meteorology |date=2023 |volume=336 |
This outcome was especially true for decisions that impacted the end user in a significant way, such as graduate school admissions. Participants judged algorithms to be too inflexible and unforgiving in comparison to human decision-makers; instead of rigidly adhering to a set of rules, humans are able to consider exceptional cases as well as appeals to their initial decision.<ref name="Explaining Decision-Making Algorithms through UI" /> For such decisions, explainability will not necessarily cause end users to accept the use of decision-making algorithms. We will need to either turn to another method to increase trust and acceptance of decision-making algorithms, or question the need to rely solely on AI for such impactful decisions in the first place.
However, some emphasize that the purpose of explainability of artificial intelligence is not to merely increase users' trust in the system's decisions, but to calibrate the users' level of trust to the correct level.<ref>{{Cite journal |last1=Liel |first1=Yotam |last2=Zalmanson |first2=Lior |date=August 2023 |title=Turning Off Your Better Judgment – Conformity to Algorithmic Recommendations |url=http://journals.aom.org/doi/full/10.5465/AMPROC.2023.277bp |journal=Academy of Management Proceedings |language=en |volume=2023 |issue=1 |article-number=16319 |doi=10.5465/AMPROC.2023.277bp |s2cid=260212999 |issn=0065-0668|url-access=subscription }}</ref> According to this principle, too much or too little user trust in the AI system will harm the overall performance of the human-system unit. When the trust is excessive, the users are not critical of possible mistakes of the system and when the users do not have enough trust in the system, they will not exhaust the benefits inherent in it.
== Criticism ==
Some scholars have suggested that explainability in AI should be considered a goal secondary to AI effectiveness, and that encouraging the exclusive development of XAI may limit the functionality of AI more broadly.<ref name=":5">{{Cite journal|last1=McCoy|first1=Liam G.|last2=Brenna|first2=Connor T. A.|last3=Chen|first3=Stacy S.|last4=Vold|first4=Karina|last5=Das|first5=Sunit|date=2021-11-05|title=Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based|url=https://www.jclinepi.com/article/S0895-4356(21)00354-1/abstract|journal=Journal of Clinical Epidemiology|volume=142|language=English|issue=Online ahead of print|pages=252–257|doi=10.1016/j.jclinepi.2021.11.001|issn=0895-4356|pmid=34748907|s2cid=243810442}}</ref><ref>{{Cite journal|last1=Ghassemi|first1=Marzyeh|last2=Oakden-Rayner|first2=Luke|last3=Beam|first3=Andrew L.|date=2021-11-01|title=The false hope of current approaches to explainable artificial intelligence in health care
Some researchers advocate the use of inherently interpretable machine learning models, rather than using post-hoc explanations in which a second model is created to explain the first. This is partly because post-hoc models increase the complexity in a decision pathway and partly because it is often unclear how faithfully a post-hoc explanation can mimic the computations of an entirely separate model.<ref name=":6" /> However, another view is that what is important is that the explanation accomplishes the given task at hand, and whether it is pre or post-hoc doesn't matter. If a post-hoc explanation method helps a doctor diagnose cancer better, it is of secondary importance whether it is a correct/incorrect explanation.
Line 114 ⟶ 127:
== Explainability in social choice ==
Explainability was studied also in [[social choice theory]]. Social choice theory aims at finding solutions to social decision problems, that are based on well-established axioms. [[Ariel D. Procaccia]]<ref>{{Citation |last=Procaccia |first=Ariel D. |title=Axioms Should Explain Solutions |date=2019 |work=The Future of Economic Design: The Continuing Development of a Field as Envisioned by Its Researchers |series=Studies in Economic Design |pages=195–199 |editor-last=Laslier |editor-first=Jean-François |url=https://doi.org/10.1007/978-3-030-18050-8_27 |access-date=2024-03-21 |place=Cham |publisher=Springer International Publishing |language=en |doi=10.1007/978-3-030-18050-8_27 |isbn=978-3-030-18050-8 |editor2-last=Moulin |editor2-first=Hervé |editor3-last=Sanver |editor3-first=M. Remzi |editor4-last=Zwicker |editor4-first=William S.|url-access=subscription }}</ref> explains that these axioms can be used to construct convincing explanations to the solutions. This principle has been used to construct explanations in various subfields of social choice.
=== Voting ===
Line 122 ⟶ 135:
=== Participatory budgeting ===
Yang, Hausladen, Peters, Pournaras, Fricker and Helbing<ref name=":12">{{Cite
=== Payoff allocation ===
Nizri, Azaria and Hazon<ref>{{Cite journal |last1=Nizri |first1=Meir |last2=Hazon |first2=Noam |last3=Azaria |first3=Amos |date=2022-06-28 |title=Explainable Shapley-Based Allocation (Student Abstract) |url=https://ojs.aaai.org/index.php/AAAI/article/view/21648 |journal=Proceedings of the AAAI Conference on Artificial Intelligence |language=en |volume=36 |issue=11 |pages=13023–13024 |doi=10.1609/aaai.v36i11.21648 |s2cid=250296641 |issn=2374-3468|doi-access=free }}</ref> present an algorithm for computing explanations for the [[Shapley value]]. Given a coalitional game, their algorithm
== See also ==
Line 140 ⟶ 153:
* {{cite web | url=https://fatconference.org/ | title=ACM Conference on Fairness, Accountability, and Transparency (FAccT) }}
* {{ Cite journal | title= Random Forest similarity maps: A Scalable Visual Representation for Global and Local Interpretation| year= 2021| doi= 10.3390/electronics10222862| doi-access= free| last1= Mazumdar| first1= Dipankar| last2= Neto| first2= Mário Popolin| last3= Paulovich| first3= Fernando V.| journal= Electronics| volume= 10| issue= 22| page= 2862}}
* {{cite web | title=Explainable AI: Making machines understandable for humans | website=Explainable AI: Making machines understandable for humans | url=https://explainableai.com/ | ref={{sfnref | Explainable AI: Making machines understandable for humans}} | access-date=2017-11-02}}
* {{cite web |title=Explaining How End-to-End Deep Learning Steers a Self-Driving Car |website=Parallel Forall |date=2017-05-23 |url=https://devblogs.nvidia.com/parallelforall/explaining-deep-learning-self-driving-car/ |ref={{sfnref | Parallel Forall | 2017}} |access-date=2017-11-02}}
Line 152 ⟶ 164:
[[Category:Quality control tools]]
[[Category:Artificial intelligence]]
[[Category:Artificial intelligence engineering]]
| |||