Content deleted Content added
Keith arano (talk | contribs) new article |
Citation bot (talk | contribs) Altered template type. Add: class, eprint. Removed URL that duplicated identifier. Removed access-date with no URL. Removed parameters. Some additions/deletions were parameter name changes. | Use this bot. Report bugs. | Suggested by Headbomb | #UCB_toolbar |
||
| (47 intermediate revisions by 26 users not shown) | |||
Line 1:
{{Short description|Technology for sentiment analysis}}
'''Multimodal
Similar to the traditional [[sentiment analysis]], one of the most basic
== Features ==
[[Feature engineering]], which involves the selection of features that are fed into [[machine learning]] algorithms, plays a key role in the
=== Textual
Similar to the conventional text-based [[sentiment analysis]], some of the most commonly used textual features in multimodal
=== Audio
[[Feeling|Sentiment]] and [[emotion]] characteristics are prominent in different [[phonetic]] and [[prosodic]] properties contained in audio features.<ref>{{cite journal |last1=Chung-Hsien Wu |last2=Wei-Bin Liang |title=Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels |journal=IEEE Transactions on Affective Computing |date=January 2011 |volume=2 |issue=1 |pages=10–21 |doi=10.1109/T-AFFC.2010.16|s2cid=52853112 }}</ref> Some of the most important audio features employed in multimodal
=== Visual features ===
▲[[Sentiment]] and [[emotion]] characteristics are prominent in different [[phonetic]] and [[prosodic]] properties contained in audio features.<ref>{{cite journal |last1=Chung-Hsien Wu |last2=Wei-Bin Liang |title=Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels |journal=IEEE Transactions on Affective Computing |date=January 2011 |volume=2 |issue=1 |pages=10–21 |doi=10.1109/T-AFFC.2010.16}}</ref> Some of the most important audio features employed in multimodal [[sentiment analysis]] are [[mel-frequency cepstrum| mel-frequency cepstrum (MFCC)]], [[spectral centroid]], [[spectral flux]], [[beat]] histogram, [[beat]] sum, strongest [[beat]], pause duration, and [[pitch]].<ref name="s1"></ref> [[OpenSMILE]]<ref>{{cite journal |last1=Eyben |first1=Florian |last2=Wöllmer |first2=Martin |last3=Schuller |first3=Björn |title=OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit - IEEE Conference Publication |journal=ieeexplore.ieee.org |date=2009 |doi=10.1109/ACII.2009.5349350 |url=http://ieeexplore.ieee.org/document/5349350}}</ref> and [[Praat]] are popular open-source toolkits for extracting such audio features.<ref>{{cite journal |last1=Morency |first1=Louis-Philippe |last2=Mihalcea |first2=Rada |last3=Doshi |first3=Payal |title=Towards multimodal sentiment analysis: harvesting opinions from the web |date=14 November 2011 |pages=169–176 |doi=10.1145/2070481.2070509 |url=https://dl.acm.org/citation.cfm?id=2070509 |publisher=ACM}}</ref>
One of the main advantages of analyzing videos
Unlike the traditional text-based [[sentiment analysis]], multimodal
▲One of the main advantages of analyzing videos over texts alone, are the presence of rich [[sentiment]] cues in visual data.<ref>{{cite journal |last1=Poria |first1=Soujanya |last2=Cambria |first2=Erik |last3=Hazarika |first3=Devamanyu |last4=Majumder |first4=Navonil |last5=Zadeh |first5=Amir |last6=Morency |first6=Louis-Philippe |title=Context-Dependent Sentiment Analysis in User-Generated Videos |journal=Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |date=2017 |doi=10.18653/v1/p17-1081 |url=https://doi.org/10.18653/v1/P17-1081 |publisher=Association for Computational Linguistics}}</ref> Visual features include [[facial expression]]s, which are principal signs of understanding [[sentiment]]s and [[emotion]]s, as they are a main channel of forming a person's present state of mind.<ref name="s1"></ref> Specifically, [[smile]], is considered to be one of the most predictive visual cues in multimodal [[sentiment analysis]].<ref name="s2"></ref> OpenFace is an open-source facial analysis toolkit available for extracting and understanding such visual features.<ref>{{cite journal |title=OpenFace: An open source facial behavior analysis toolkit - IEEE Conference Publication |journal=ieeexplore.ieee.org |url=https://ieeexplore.ieee.org/document/7477553/}}</ref>
Feature-level fusion (sometimes known as early fusion)
===
Decision-level fusion (sometimes known as late fusion), feeds data from each
▲Unlike the traditional text-based [[sentiment analysis]], multimodal [[sentiment analysis]] undergoes a fusion process in which data from different [[Modality (human–computer interaction)|modalities]] (text, audio, or visual) are fused and analyzed together.<ref name ="s1"></ref> The existing approaches in multimodal [[sentiment analysis]] [[data fusion]] can be grouped into three main categories: feature-level, decision-level, and hybrid fusion, and the performance of the [[sentiment]] classification depends on which type of fusion technique is employed.<ref name ="s1"></ref>
Hybrid fusion is a combination of feature-level and decision-level fusion techniques, which exploits complementary information from both methods during the classification process.<ref name="s4"
▲=== Feature-level Fusion ===
▲Feature-level fusion (sometimes known as early fusion), gathers all features from each [[modality (human–computer interaction)|modality]] (text, audio, or visual) and joins them together into a single feature vector, which is eventually fed into a classification algorithm<ref name="s3">{{cite journal |last1=Poria |first1=Soujanya |last2=Cambria |first2=Erik |last3=Howard |first3=Newton |last4=Huang |first4=Guang-Bin |last5=Hussain |first5=Amir |title=Fusing audio, visual and textual clues for sentiment analysis from multimodal content |journal=Neurocomputing |date=January 2016 |volume=174 |pages=50–59 |doi=10.1016/j.neucom.2015.01.095}}</ref>. One of the difficulties in implementing this technique is the integration of the heterogeneous features.<ref name="s1"></ref>
▲Decision-level fusion (sometimes known as late fusion), feeds data from each [[modality (human–computer interaction)|modality]] (text, audio, or visual) independently into its own classification algorithm, and obtains the final [[sentiment]] classification results by fusing each result into a single decision vector.<ref name="s3"></ref> One of the advantages of this fusion technique is it eliminates the need to fuse heterogeneous data, and each [[modality (human–computer interaction)|modality]] can utilize its most appropriate classification algorithm.<ref name="s1"></ref>
▲=== Hybrid Fusion ===
▲Hybrid fusion is a combination of feature-level and decision-level fusion techniques, which exploits complementary information from both methods during the classification process.<ref name="s4"></ref> It usually involves a two-step procedure wherein feature-level fusion is initially performed between two [[Modality (human–computer interaction)|modalities]], and decision-level fusion is then applied as a second step, to fuse the initial results from the feature-level fusion, with the remaining [[Modality (human–computer interaction)|modality]].<ref>{{cite journal |last1=Shahla |first1=Shahla |last2=Naghsh-Nilchi |first2=Ahmad Reza |title=Exploiting evidential theory in the fusion of textual, audio, and visual modalities for affective music video retrieval - IEEE Conference Publication |journal=ieeexplore.ieee.org |date=2017 |url=https://ieeexplore.ieee.org/abstract/document/7983051/}}</ref><ref>{{cite journal |last1=Poria |first1=Soujanya |last2=Peng |first2=Haiyun |last3=Hussain |first3=Amir |last4=Howard |first4=Newton |last5=Cambria |first5=Erik |title=Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis |journal=Neurocomputing |date=October 2017 |volume=261 |pages=217–230 |doi=10.1016/j.neucom.2016.09.117}}</ref>
== Applications ==
Similar to text-based
==References==▼
▲Similar to text-based [[sentiment analysis]], multimodal [[sentiment analysis]] can be applied in the development of different forms of [[recommender system]]s such as in the analysis of user-generated videos of movie reviews<ref name="s4"></ref> and general product reviews<ref>{{cite journal |last1=Pérez-Rosas |first1=Verónica |last2=Mihalcea |first2=Rada |last3=Morency |first3=Louis Philippe |title=Utterance-level multimodal sentiment analysis |journal=Long Papers |date=1 January 2013 |url=https://experts.umich.edu/en/publications/utterance-level-multimodal-sentiment-analysis |publisher=Association for Computational Linguistics (ACL)}}</ref>, to predict the [[sentiment]]s of customers, and subsequently create product or service recommendations.<ref>{{cite web |last1=Chui |first1=Michael |last2=Manyika |first2=James |last3=Miremadi |first3=Mehdi |last4=Henke |first4=Nicolaus |last5=Chung |first5=Rita |last6=Nel |first6=Pieter |last7=Malhotra |first7=Sankalp |title=Notes from the AI frontier. Insights from hundreds of use cases |url=https://www.mckinsey.com/mgi/ |website=McKinsey & Company |publisher=McKinsey & Company |accessdate=13 June 2018 |language=en}}</ref> Multimodal [[sentiment analysis]] also plays an important role in the advancement of [[virtual assistant]]s through the application of [[natural language processing]] (NLP) and [[machine learning]] techniques.<ref name ="s5"></ref> In the healthcare ___domain, multimodal [[sentiment analysis]] can be utilized to detect certain medical conditions such as [[Psychological stress|stress]], [[anxiety]], or [[Depression (mood)|depression]].<ref name = "s6"></ref> Multimodal [[sentiment analysis]] can also be applied in understanding the [[sentiment]]s contained in video news programs, which is considered as a complicated and challenging ___domain, as sentiments expressed by reporters tend to be less obvious or neutral.<ref>{{cite journal |last1=Ellis |first1=Joseph G. |last2=Jou |first2=Brendan |last3=Chang |first3=Shih-Fu |title=Why We Watch the News: A Dataset for Exploring Sentiment in Broadcast Video News |date=12 November 2014 |pages=104–111 |doi=10.1145/2663204.2663237 |url=https://dl.acm.org/citation.cfm?doid=2663204.2663237 |publisher=ACM}}</ref>
{{Reflist}}
▲==References==
[[Category:Natural language processing]]
[[Category:Affective computing]]
[[Category:Social media]]
[[Category:Machine learning]]
[[Category:Multimodal interaction]]
| |||