Content deleted Content added
https://en.wikipedia.org/w/index.php?title=User:Yasmeenbg/Explainable_artificial_intelligence&action=edit |
Citation bot (talk | contribs) Alter: date, doi, title, template type. Add: pages, issue, journal, arxiv, doi, volume, series, doi-broken-date, authors 1-1. Removed parameters. Some additions/deletions were parameter name changes. Upgrade ISBN10 to 13. | Use this bot. Report bugs. | Suggested by Headbomb | #UCB_toolbar |
||
Line 3:
{{artificial intelligence}}
'''Explainable AI''' ('''XAI'''), often overlapping with '''interpretable AI''', or '''explainable machine learning''' ('''XML'''), either refers to an [[artificial intelligence]] (AI) system over which it is possible for humans to retain ''intellectual oversight'', or refers to the methods to achieve this.<ref>{{Cite journal|last=Longo|first=Luca|display-authors=etal|date=2024 |title=Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions|url=https://www.sciencedirect.com/science/article/pii/S1566253524000794 |journal=Information Fusion|volume=106|doi=10.1016/j.inffus.2024.102301}}</ref><ref>{{Cite journal |last=Mihály |first=Héder |date=2023 |title=Explainable AI: A Brief History of the Concept |url=https://ercim-news.ercim.eu/images/stories/EN134/EN134-web.pdf |journal=ERCIM News |issue=134 |pages=9–10}}</ref> The main focus is usually on the reasoning behind the decisions or predictions made by the AI<ref>{{Cite journal |last1=Phillips |first1=P. Jonathon |last2=Hahn |first2=Carina A. |last3=Fontana |first3=Peter C. |last4=Yates |first4=Amy N. |last5=Greene |first5=Kristen |last6=Broniatowski |first6=David A. |last7=Przybocki |first7=Mark A. |date=2021-09-29 |title=Four Principles of Explainable Artificial Intelligence |url=https://doi.org/10.6028/NIST.IR.8312 |journal=NIST |doi=10.6028/nist.ir.8312}}</ref> which are made more understandable and transparent.<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title=Notions of explainability and evaluation approaches for explainable artificial intelligence|url=https://www.sciencedirect.com/science/article/pii/S1566253521001093|journal=Information Fusion|year=2021|volume= December 2021 - Volume 76 |pages=89–106|doi=10.1016/j.inffus.2021.05.009}}</ref> This has been brought up again as a topic of active research as users now need to know the safety and explain what automated decision making is in different applications.<ref>{{Cite journal |
XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason.<ref>{{Cite journal|last=Alizadeh|first=Fatemeh|date=2021|title=I Don't Know, Is AI Also Used in Airbags?: An Empirical Study of Folk Concepts and People's Expectations of Current and Future Artificial Intelligence|url=https://www.researchgate.net/publication/352638184|journal=Icom|volume=20 |issue=1 |pages=3–17 |doi=10.1515/icom-2021-0009|doi-broken-date=3 December 2024 |s2cid=233328352}}</ref> XAI may be an implementation of the social [[right to explanation]].<ref name=":0">{{Cite journal|last1=Edwards|first1=Lilian|last2=Veale|first2=Michael|date=2017|title=Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For|journal=Duke Law and Technology Review|volume=16|pages=18|ssrn=2972855}}</ref> Even if there is no such legal right or regulatory requirement, XAI can improve the [[user experience]] of a product or service by helping end users trust that the AI is making good decisions.<ref>{{Cite web |last=Do Couto |first=Mark |date=February 22, 2024 |title=Entering the Age of Explainable AI |url=https://tdwi.org/Articles/2024/02/22/ADV-ALL-Entering-the-Age-of-Explainable-AI.aspx |access-date=2024-09-11 |website=TDWI}}</ref> XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on.<ref name=":3">{{Cite journal|last1=Gunning|first1=D.|last2=Stefik|first2=M.|last3=Choi|first3=J.|last4=Miller|first4=T.|last5=Stumpf|first5=S.|last6=Yang|first6=G.-Z.|date=2019-12-18|title=XAI-Explainable artificial intelligence|url=https://openaccess.city.ac.uk/id/eprint/23405/|journal=Science Robotics|language=en|volume=4|issue=37|pages=eaay7120|doi=10.1126/scirobotics.aay7120|pmid=33137719|issn=2470-9476|doi-access=free}}</ref> This makes it possible to confirm existing knowledge, challenge existing knowledge, and generate new assumptions.<ref>{{Cite journal|last1=Rieg|first1=Thilo|last2=Frick|first2=Janek|last3=Baumgartl|first3=Hermann|last4=Buettner|first4=Ricardo|date=2020-12-17|title=Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms|journal=PLOS ONE|language=en|volume=15|issue=12|pages=e0243615|doi=10.1371/journal.pone.0243615|issn=1932-6203|pmc=7746264|pmid=33332440|bibcode=2020PLoSO..1543615R|doi-access=free}}</ref>
[[Machine learning]] (ML) algorithms used in AI can be categorized as [[White-box testing|white-box]] or [[Black box|black-box]].<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title= Classification of Explainable Artificial Intelligence Methods through Their Output Formats |journal=Machine Learning and Knowledge Extraction|year=2021|volume=3|issue=3|pages=615–661|doi=10.3390/make3030032|doi-access=free }}</ref> White-box models provide results that are understandable to experts in the ___domain. Black-box models, on the other hand, are extremely hard to explain and may not be understood even by ___domain experts.<ref>{{Cite journal|last=Loyola-González|first=O.|date=2019|title=Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View|journal=IEEE Access|volume=7|pages=154096–154113|doi=10.1109/ACCESS.2019.2949286|bibcode=2019IEEEA...7o4096L |issn=2169-3536|doi-access=free}}</ref> XAI algorithms follow the three principles of transparency, interpretability, and explainability. A model is transparent "if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer."<ref name=":4">{{Cite journal|last1=Roscher|first1=R.|last2=Bohn|first2=B.|last3=Duarte|first3=M. F.|last4=Garcke|first4=J.|date=2020|title=Explainable Machine Learning for Scientific Insights and Discoveries|journal=IEEE Access|volume=8|pages=42200–42216|doi=10.1109/ACCESS.2020.2976199|arxiv=1905.08883 |bibcode=2020IEEEA...842200R |issn=2169-3536|doi-access=free}}</ref> Interpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.<ref name="Interpretable machine learning: def">{{cite journal|last1=Murdoch|first1=W. James|last2=Singh|first2=Chandan|last3=Kumbier|first3=Karl|last4=Abbasi-Asl|first4=Reza|last5=Yu|first5=Bin|date=2019-01-14|title=Interpretable machine learning: definitions, methods, and applications|journal=Proceedings of the National Academy of Sciences of the United States of America|volume=116|issue=44|pages=22071–22080|arxiv=1901.04592|doi=10.1073/pnas.1900654116|pmid=31619572|pmc=6825274|bibcode= |doi-access=free}}</ref><ref name="Lipton 31–57">{{Cite journal|last=Lipton|first=Zachary C.|date=June 2018|title=The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.|journal=Queue|language=en|volume=16|issue=3|pages=31–57|doi=10.1145/3236386.3241340|issn=1542-7730|doi-access=free}}</ref><ref>{{Cite web|date=2019-10-22|title=Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI|url=https://deepai.org/publication/explainable-artificial-intelligence-xai-concepts-taxonomies-opportunities-and-challenges-toward-responsible-ai|access-date=2021-01-13|website=DeepAI}}</ref> Explainability is a concept that is recognized as important, but a consensus definition is not yet available;<ref name=":4" /> one possibility is "the collection of features of the interpretable ___domain that have contributed, for a given example, to producing a decision (e.g., classification or regression)".<ref>{{Cite journal|date=2018-02-01|title=Methods for interpreting and understanding deep neural networks|journal=Digital Signal Processing|language=en|volume=73|pages=1–15|doi=10.1016/j.dsp.2017.10.011|issn=1051-2004|doi-access=free|last1=Montavon|first1=Grégoire|last2=Samek|first2=Wojciech|last3=Müller|first3=Klaus-Robert|arxiv=1706.07979 |bibcode=2018DSP....73....1M |author-link3=Klaus-Robert Müller}}</ref>
In summary, Interpretability refers to the user's ability to understand model outputs, while Model Transparency includes Simulatability (reproducibility of predictions), Decomposability (intuitive explanations for parameters), and Algorithmic Transparency (explaining how algorithms work). Model Functionality focuses on textual descriptions, visualization, and local explanations, which clarify specific outputs or instances rather than entire models. All these concepts aim to enhance the comprehensibility and usability of AI systems.<ref name="NCB23">{{Cite web|author = Notovich, Aviv., Chalutz-Ben Gal, Hila, & Ben-Gal, Irad (2023).
If algorithms fulfill these principles, they provide a basis for justifying decisions, tracking them and thereby verifying them, improving the algorithms, and exploring new facts.<ref>{{Cite journal|last1=Adadi|first1=A.|last2=Berrada|first2=M.|date=2018|title=Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)|journal=IEEE Access|volume=6|pages=52138–52160|doi=10.1109/ACCESS.2018.2870052|bibcode=2018IEEEA...652138A |issn=2169-3536|doi-access=free}}</ref>
Line 47:
For images, [[Saliency map|saliency maps]] highlight the parts of an image that most influenced the result.<ref>{{Cite web |last=Sharma |first=Abhishek |date=2018-07-11 |title=What Are Saliency Maps In Deep Learning? |url=https://analyticsindiamag.com/what-are-saliency-maps-in-deep-learning/ |access-date=2024-07-10 |website=Analytics India Magazine |language=en-US}}</ref>
Systems that are expert or knowledge based are software systems that are made my experts. This system consists of a knowledge based encoding for the ___domain knowledge. This system is usually modeled as production rules, and someone uses this knowledge base which the user can question the system for knowledge. In expert systems, the language and explanations are understood with an explanation for the reasoning or a problem solving activity.<ref>{{Cite journal |
However, these techniques are not very suitable for [[Language model|language models]] like [[Generative pre-trained transformer|generative pretrained transformers]]. Since these models generate language, they can provide an explanation, but which may not be reliable. Other techniques include attention analysis (examining how the model focuses on different parts of the input), probing methods (testing what information is captured in the model's representations), causal tracing (tracing the flow of information through the model) and circuit discovery (identifying specific subnetworks responsible for certain behaviors). Explainability research in this area overlaps significantly with interpretability and [[AI alignment|alignment]] research.<ref>{{cite arXiv |last1=Luo |first1=Haoyan |title=From Understanding to Utilization: A Survey on Explainability for Large Language Models |date=2024-02-21 |eprint=2401.12874 |last2=Specia |first2=Lucia|class=cs.CL }}</ref>
Line 73:
[[Marvin Minsky]] et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.<ref>Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.</ref>
Explainable AI has been recently a new topic researched amongst the context of modern deep learning. Modern complex AI techniques, such as [[deep learning]], are naturally opaque.<ref>{{cite magazine|last1=Mukherjee|first1=Siddhartha|title=A.I. Versus M.D.|url=https://www.newyorker.com/magazine/2017/04/03/ai-versus-md|access-date=30 January 2018|magazine=The New Yorker|date=27 March 2017}}</ref> To address this issue, methods have been developed to make new models more explainable and interpretable.<ref>{{Cite journal|date=2020-07-08|title=Interpretable neural networks based on continuous-valued logic and multicriteria decision operators|journal=Knowledge-Based Systems|language=en|volume=199|pages=105972|doi=10.1016/j.knosys.2020.105972 |arxiv=1910.02486 |issn=0950-7051|doi-access=free|last1=Csiszár|first1=Orsolya|last2=Csiszár|first2=Gábor|last3=Dombi|first3=József}}</ref><ref name="Lipton 31–57"/><ref name="Interpretable machine learning: def"/><ref>{{cite arXiv|last1=Doshi-Velez|first1=Finale|last2=Kim|first2=Been|date=2017-02-27|title=Towards A Rigorous Science of Interpretable Machine Learning|eprint=1702.08608|class=stat.ML}}</ref><ref>{{Cite arXiv |last=Abdollahi, Behnoush, and Olfa Nasraoui.|title=Explainable Restricted Boltzmann Machines for Collaborative Filtering.|eprint=1606.07129|class=stat.ML|year=2016}}</ref><ref>{{Cite book|last1=Dombi|first1=József|last2=Csiszár|first2=Orsolya|series=Studies in Fuzziness and Soft Computing |date=2021|title=Explainable Neural Networks Based on Fuzzy Logic and Multi-criteria Decision Tools|url=https://link.springer.com/book/10.1007/978-3-030-72280-7|volume=408|language=en-gb|doi=10.1007/978-3-030-72280-7|isbn=978-3-030-72279-1|s2cid=233486978|issn=1434-9922}}</ref> This includes layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output.<ref name="Bach Binder Montavon Klauschen p=e0130140">{{cite journal|last1=Bach|first1=Sebastian|last2=Binder|first2=Alexander|last3=Montavon|first3=Grégoire|last4=Klauschen|first4=Frederick|last5=Müller|first5=Klaus-Robert|author-link5=Klaus-Robert Müller|last6=Samek|first6=Wojciech|date=2015-07-10|editor-last=Suarez|editor-first=Oscar Deniz|title=On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation|journal=PLOS ONE|volume=10|issue=7|page=e0130140|bibcode=2015PLoSO..1030140B|doi=10.1371/journal.pone.0130140|issn=1932-6203|pmc=4498753|pmid=26161953|doi-access=free}}</ref><ref>{{cite news|url=https://www.theguardian.com/science/2017/nov/05/computer-says-no-why-making-ais-fair-accountable-and-transparent-is-crucial|title=Computer says no: why making AIs fair, accountable and transparent is crucial|last1=Sample|first1=Ian|date=5 November 2017|work=The Guardian|access-date=5 August 2018|language=en}}</ref> Other techniques explain some particular prediction made by a (nonlinear) black-box model, a goal referred to as "local interpretability".<ref>{{Cite journal|last1=Martens|first1=David|last2=Provost|first2=Foster|title=Explaining data-driven document classifications|url=http://pages.stern.nyu.edu/~fprovost/Papers/MartensProvost_Explaining.pdf|journal=MIS Quarterly|year=2014|volume=38|pages=73–99|doi=10.25300/MISQ/2014/38.1.04|s2cid=14238842}}</ref><ref>{{Cite journal|title="Why Should I Trust You?" {{!}} Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining|language=EN|doi=10.1145/2939672.2939778|s2cid=13029170}}</ref><ref>{{Citation|last1=Lundberg|first1=Scott M|title=A Unified Approach to Interpreting Model Predictions|date=2017|url=http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf|work=Advances in Neural Information Processing Systems 30|pages=4765–4774|editor-last=Guyon|editor-first=I.|publisher=Curran Associates, Inc.|access-date=2020-03-13|last2=Lee|first2=Su-In|editor2-last=Luxburg|editor2-first=U. V.|editor3-last=Bengio|editor3-first=S.|editor4-last=Wallach|editor4-first=H.|arxiv=1705.07874}}</ref><ref>{{Cite journal|last1=Carter|first1=Brandon|last2=Mueller|first2=Jonas|last3=Jain|first3=Siddhartha|last4=Gifford|first4=David|date=2019-04-11|title=What made you do this? Understanding black-box decisions with sufficient input subsets|url=http://proceedings.mlr.press/v89/carter19a.html|journal=The 22nd International Conference on Artificial Intelligence and Statistics|language=en|pages=567–576}}</ref><ref>{{Cite journal|last1=Shrikumar|first1=Avanti|last2=Greenside|first2=Peyton|last3=Kundaje|first3=Anshul|date=2017-07-17|title=Learning Important Features Through Propagating Activation Differences|url=http://proceedings.mlr.press/v70/shrikumar17a.html|journal=International Conference on Machine Learning|language=en|pages=3145–3153}}</ref><ref>{{Cite journal|url=https://dl.acm.org/doi/abs/10.5555/3305890.3306024|title=Axiomatic attribution for deep networks {{!}} Proceedings of the 34th International Conference on Machine Learning - Volume 70|website=dl.acm.org|series=Icml'17|date=6 August 2017|pages=3319–3328|language=EN|access-date=2020-03-13}}</ref> We still today cannot explain the output of today's DNNs without the new explanatory mechanisms, we also can't by the neural network, or external explanatory components <ref>{{Citation |
There has been work on making glass-box models which are more transparent to inspection.<ref name=":6"/><ref>{{cite journal |last1=Singh |first1=Chandan |last2=Nasseri |first2=Keyan |last3=Tan |first3=Yan Shuo |last4=Tang |first4=Tiffany |last5=Yu |first5=Bin |title=imodels: a python package for fitting interpretable models |journal=Journal of Open Source Software |date=4 May 2021 |volume=6 |issue=61 |pages=3192 |doi=10.21105/joss.03192 |bibcode=2021JOSS....6.3192S |s2cid=235529515 |url=https://joss.theoj.org/papers/10.21105/joss.03192 |language=en |issn=2475-9066}}</ref> This includes [[decision tree]]s,<ref>{{Cite journal|last1=Vidal|first1=Thibaut|last2=Schiffer|first2=Maximilian|date=2020|title=Born-Again Tree Ensembles|url=http://proceedings.mlr.press/v119/vidal20a.html|journal=International Conference on Machine Learning|language=en|publisher=PMLR|volume=119|pages=9743–9753|arxiv=2003.11132}}</ref> [[Bayesian network]]s, sparse [[linear model]]s,<ref>{{cite journal |last1=Ustun |first1=Berk |last2=Rudin |first2=Cynthia |title=Supersparse linear integer models for optimized medical scoring systems |journal=Machine Learning |date=1 March 2016 |volume=102 |issue=3 |pages=349–391 |doi=10.1007/s10994-015-5528-6 |s2cid=207211836 |url=https://link.springer.com/article/10.1007/s10994-015-5528-6 |language=en |issn=1573-0565}}</ref> and more.<ref>Bostrom, N., & Yudkowsky, E. (2014). [https://intelligence.org/files/EthicsofAI.pdf The ethics of artificial intelligence]. ''The Cambridge Handbook of Artificial Intelligence'', 316-334.</ref> The [[ACM Conference on Fairness, Accountability, and Transparency|Association for Computing Machinery Conference on Fairness, Accountability, and Transparency (ACM FAccT)]] was established in 2018 to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.<ref name="FAT* conference">{{cite web | url=https://fatconference.org/ | title=FAT* Conference }}</ref><ref>{{cite news |title=Computer programs recognise white men better than black women |url=https://www.economist.com/science-and-technology/2018/02/15/computer-programs-recognise-white-men-better-than-black-women |access-date=5 August 2018 |newspaper=The Economist |date=2018 |language=en}}</ref>
Line 85:
== Regulation ==
As regulators, official bodies, and general users come to depend on AI-based dynamic systems, clearer accountability will be required for [[automated decision-making]] processes to ensure trust and transparency. The first global conference exclusively dedicated to this emerging discipline was the 2017 [[International Joint Conference on Artificial Intelligence]]: Workshop on Explainable Artificial Intelligence (XAI).<ref>{{cite web|title=IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)|url=http://www.intelligentrobots.org/files/IJCAI2017/IJCAI-17_XAI_WS_Proceedings.pdf|website=Earthlink|publisher=IJCAI|access-date=17 July 2017|archive-date=4 April 2019|archive-url=https://web.archive.org/web/20190404131609/http://www.intelligentrobots.org/files/IJCAI2017/IJCAI-17_XAI_WS_Proceedings.pdf|url-status=dead}}</ref> It has evolved over the years, with various workshops organised and co-located to many other international conferences, and it has now a dedicated global event, "The world conference on eXplainable Artificial Intelligence", with its own proceedings.<ref name="XAI-2023">{{cite book |author=<!--Not stated--> |date= 2023| title= Explainable Artificial Intelligence, First World Conference, xAI 2023, Lisbon, Portugal, July 26–28, 2023, Proceedings, Parts I/II/III |series= Communications in Computer and Information Science|volume= 1903|url= https://link.springer.com/book/10.1007/978-3-031-44070-0 |publisher=springer |doi= 10.1007/978-3-031-44070-0|isbn=978-3-031-44070-0}}</ref><ref name="XAI-2024">{{cite book |author=<!--Not stated-->|date= 2024| title= Explainable Artificial Intelligence, Second World Conference, xAI 2024, Valletta, Malta, July 17–19, 2024, Proceedings, Part I/II/III/IV |series= Communications in Computer and Information Science|volume= 2153|url=https://link.springer.com/book/10.1007/978-3-031-63787-2 |publisher=springer |doi= 10.1007/978-3-031-63787-2|isbn=978-3-031-63787-2}}</ref>
Line 96:
By making an AI system more explainable, we also reveal more of its inner workings. For example, the explainability method of feature importance identifies features or variables that are most important in determining the model's output, while the influential samples method identifies the training samples that are most influential in determining the output, given a particular input.<ref name="Explainable Machine Learning in Deployment">{{cite book | last1=Bhatt | first1=Umang | last2=Xiang | first2=Alice | last3=Sharma | first3=Shubham | last4=Weller | first4=Adrian | last5=Taly | first5=Ankur | last6=Jia | first6=Yunhan | last7=Ghosh | first7=Joydeep | last8=Puri | first8=Richir | last9=M.F. Moura | first9=José | last10=Eckersley | first10=Peter | title=Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency | chapter=Explainable Machine Learning in Deployment | date=2022 | pages=648–657 | doi=10.1145/3351095.3375624 | isbn=9781450369367 | s2cid=202572724 | chapter-url=https://dl.acm.org/doi/pdf/10.1145/3351095.3375624 }}</ref> Adversarial parties could take advantage of this knowledge.
For example, competitor firms could replicate aspects of the original AI system in their own product, thus reducing competitive advantage.<ref name="How the machine 'thinks'">{{cite journal |last1=Burrel |first1=Jenna |date=2016 |title=How the machine 'thinks': Understanding opacity in machine learning algorithms |url=https://journals.sagepub.com/doi/pdf/10.1177/2053951715622512 |journal
=== '''Adaptive Integration and Explanation''' ===
Many approaches that it uses provides explanation in general, it doesn't take account for the diverse backgrounds and knowledge level of the users. This leads to challenges with accurate comprehension for all users. Expert users can find the explanations lacking in depth, and are oversimplified, while a beginner user may struggle understanding the explanations as they are complex. This limitation downplays the ability of the XAI techniques to appeal to their users with different levels of knowledge, which can impact the trust from users and who uses it. The quality of explanations can be different amongst their users as they all have different expertise levels, including different situation and conditions<ref>{{Cite journal |
=== Technical complexity ===
Line 129:
=== Participatory budgeting ===
Yang, Hausladen, Peters, Pournaras, Fricker and Helbing<ref name=":12">{{Cite
=== Payoff allocation ===
| |||