Complex event processing: Difference between revisions

Revision as of 16:28, 22 December 2017

Latest revision as of 11:06, 23 June 2025

(38 intermediate revisions by 27 users not shown)

Line 1:

{{Short description|Method of analysing information about events}}

{{Refimprove|date=March 2010}}

'''Event processing''' is a method of tracking and [[data analytics|analyzing]] (processing) streams of information (data) about things that happen (events),<ref name=LuckhamD>{{cite book|last=Luckham|first=David C.|title=Event Processing for Business: Organizing the Real-Time Enterprise|url=http://ee.stanford.edu/~luckham/|publisher=John Wiley & Sons, Inc.,|___location = Hoboken, New Jersey|year=2012|page=3|isbn=978-0-470-53485-4}}</ref> and deriving a conclusion from them. '''Complex event processing''', or ('''CEP''',) isconsists eventof processinga thatset combinesof dataconcepts fromand multipletechniques sources<ref>{{citation|last=Schmerken|first=Ivy|title=Decipheringdeveloped in the Mythsearly Around1990s Complexfor Eventprocessing Processing|url=http://www.wallstreetandtech.com/datareal-latency/207800335|publisher=Walltime Streetevents &and Technology|date=Mayextracting 15,information 2008}}</ref>from toevent inferstreams events or patterns that suggest moreas complicatedthey circumstancesarrive. The goal of complex event processing is to identify meaningful events (such as [[business opportunity|opportunities]] or threats)<ref name=Bates>{{citation|last=Bates|first=John|title=John Bates of Progress explains how complex event processing works and how it can simplify the use of algorithms for finding and capturing trading opportunities|date=15 June 2011|url=http://fixglobal.com/home/secrets-revealed-trading-tools-uncover-hidden-opportunities/|publisher=Fix Global Trading|accessdateaccess-date=May 14, 2012}}</ref> in real-time situations and respond to them as quickly as possible.

These events may be happening across the various layers of an organization as sales leads, orders or [[customer service]] calls. Or, they may be news items,<ref>{{citation|last=Crosman|first=Penny|title=Aleri, Ravenpack to Feed News into Trading Algos|url=http://www.wallstreetandtech.com/articles/217500395|publisher=Wall Street & Technology|date=May 18, 2009}}{{Dead link|date=July 2019 |bot=InternetArchiveBot |fix-attempted=yes }}</ref> [[text messagesmessage]]s, [[social media]] [[Posting style|posts]], stockbusiness marketprocesses feeds(such as supply chain), [[traffic reportsreport]]s, [[weather forecasting|weather reportsreport]]s, or other kinds of data.<ref name=LuckhamD /> An event may also be defined as a "change of state," when a measurement exceeds a predefined threshold of time, temperature, or other value.

Analysts suggesthave suggested that CEP will give organizations a new way to analyze patterns in real-time and help the business side communicate better with IT and service departments.<ref>{{citation|last=McKay|first=Lauren|title=Forrester Gives a Welcoming Wave to Complex Event Processing|url=http://www.destinationcrm.com/Articles/CRM-News/Daily-News/Forrester-Gives-a-Welcoming-Wave-to-Complex-Event-Processing-55492.aspx|publisher=Destination CRM|date=August 13, 2009}}</ref> CEP has since become an enabling technology in many systems that are used to take immediate action in response to incoming streams of events. Applications are now to be found (2018) in many sectors of business including stock market trading systems, [[mobile device]]s, internet operations, [[fraud detection]], the [[transportation industry]], and [[Intelligence gathering network|governmental intelligence gathering]].

The vast amount of information available about events is sometimes referred to as the event cloud.<ref name=LuckhamD />

Line 19 ⟶ 21:

* Event-[[pattern detection]]

* Event [[Abstraction (computer science)|abstraction]]

* Event filtering

* Event aggregation and transformation

Line 26 ⟶ 28:

* Abstracting [[event-driven programming|event-driven]] processes

Commercial applications of CEP exist in variety of industries and include [[Algorithmic trading|algorithmic stock-trading]],<ref>[http://fixglobal.com/home/secrets-revealed-trading-tools-uncover-hidden-opportunities/ Complex Event Processing for Trading], FIXGlobal, June 2011</ref> the detection of [[credit card fraud|credit-card fraud]], [[business activity monitoring]], and security monitoring.<ref>

[http://www.complexevents.com Details of commercial products and use cases]</ref>

==History==

The CEP area has roots in [[discrete event simulation]], the [[active database]] area and some programming languages. The activity in the industry was preceded by a wave of research projects in the 1990s. According to<ref>{{citation|last=Leavit|first=Neal|title=Complex-Event Processing Poised for Growth|url= http://www.computer.org/csdl/mags/co/2009/04/mco2009040017-abs.html|publisher=Computer, vol. 42, no. 4, pp. 17-20 Washington|date=April 2009}}</ref> the first project that paved the way to a generic CEP language and execution model was the Rapide project in [[Stanford University]], directed by [[David Luckham]]. In parallel there have been two other research projects: Infospheres in [[California Institute of Technology]], directed by [[K. Mani Chandy]], and [[Apama (software)|Apama]] in [[University of Cambridge]] directed by John Bates. The commercial products were dependents of the concepts developed in these and some later research projects. Community efforts started in a series of event processing symposiumssymposia organized by the [[Event Processing Technical Society]], and later by the ACM DEBS conference series. One of the community efforts was to produce the event processing manifesto.<ref>[{{cite journal|url=http://drops.dagstuhl.de/opus/volltexte/2011/2985/|title=10201 Executive Summary and Manifesto – Event Processing|first1=Mani K. |last1=Chandy|first2=Opher|last2=Etzion|first3=Rainer andvon|last3=Ammon|journal=Drops-Idn/V2/Document/10.4230/Dagsemproc.10201.1 Opher|series=Dagstuhl EtzionSeminar andProceedings|editor-first1=K. Mani|editor-last1=Chandy|editor-first2=Opher|editor-last2=Etzion|editor-first3=Rainer von |editor-last3=Ammon(eds),|date=22 December 2017|volume=10201 Executive|pages=1–60 Summary|publisher=Schloss andDagstuhl Manifesto- Leibniz--Zentrum Eventfuer ProcessingInformatik, DagstuhlGermany|doi=10.4230/DagSemProc.10201.1 seminar|doi-access=free Procesdings|via=Dagstuhl 10201,Research ISSNOnline 1862-4405,Publication 2011]Server}}</ref>

==Related concepts==

CEP is used in [[operational intelligence]] (OI) products to provide insight into business operations by running query analysis against live feeds and event data. OI collects real-time data and correlates against historical data to provide insight and analysis. Multiple sources of data can be combined to provide a common operating picture that uses current information.

In [[network management]], [[systems management]], [[application management]] and [[service management]], people usually refer instead to [[event correlation]]. As CEP engines, event correlation engines (''event correlators'') analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events.<ref>J.P. Martin-Flatin, G. Jakobson and L. Lewis, "Event Correlation in Integrated Management: Lessons Learned and Outlook", Journal of Network and Systems Management, Vol. 17, No. 4, December 2007.</ref>

Line 59 ⟶ 61:

Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.

For example, customer service centers are using CEP for click-stream analysis and customer experience management. CEP software can factor real-time information about millions of events (clicks or other interactions) per second into [[business intelligence]] and other [[decision-support]] applications. These "[[recommendation applicationsapplication]]s" help agents provide personalized service based on each customer's experience. The CEP application may collect data about what customers on the phone are currently doing, or how they have recently interacted with the company in other various channels, including in-branch, or on the Web via self-service features, instant messaging and email. The application then analyzes the total customer experience and recommends scripts or next steps that guide the agent on the phone, and hopefully keep the customer happy.<ref>{{citation|last=Kobielus|first=James|title=Really Happy in Real Time|url=http://www.destinationcrm.com/Articles/Columns-Departments/Connect/Really-Happy-in-Real-Time-50530.aspx|publisher=Destination CRM|date=September 2008}}</ref>

==In financial services==

The financial services industry was an early adopter of CEP technology, using complex event processing to structure and contextualize available data so that it could inform trading behavior, specifically [[algorithmic trading]], by identifying opportunities or threats that indicate traders (or automatic trading systems) should buy or sell.<ref>{{citation|title=The Rise of Unstructured Data in Trading|url=http://aitegroup.com/report/rise-unstructured-data-trading|publisher=Aite Group|date=October 29, 2008}}</ref> For example, if a trader wants to track stocks that have five up movements followed by four down movements, CEP technology can track such an event. CEP technology can also track drastic rise and fall in number of trades. Algorithmic trading is already a practice in stock trading. It is estimated that around 60% of Equity trading in the United States is by way of algorithmic trades. CEP is expected to continue to help financial institutions improve their algorithms and be more efficient.

Recent improvements in CEP technologies have made it more affordable, helping smaller firms to create trading algorithms of their own and compete with larger firms.<ref name=Bates /> CEP has evolved from an emerging technology to an essential platform of many capital markets. The technology's most consistent growth has been in banking, serving fraud detection, online banking, and multichannel marketing initiatives.<ref>{{citation|title=Complex Event Processing: Beyond Capital Markets|url=http://www.aitegroup.com/Reports/ReportDetail.aspx?recordItemID=870|publisher=Aite Group|date=November 16, 2011}}</ref>

Today, a wide variety of financial applications use CEP, including profit, loss, and [[Financial risk management|risk management]] systems, [[Order book (trading)|order]] and [[Market liquidity|liquidity]] analysis, [[Quantitative investing|quantitative trading]] and signal generation systems, and others.

==Integration with time series databases==

A [[time series database]] is a software system that is optimized for the handling of data organized by time. Time series are finite or infinite sequences of data items, where each item has an associated timestamp and the sequence of timestamps is non-decreasing. Elements of a time series are often called ticks. The timestamps are not required to be ascending (merely non-decreasing) because in practice the time resolution of some systems such as financial data sources can be quite low (milliseconds, microseconds or even nanoseconds), so consecutive events may carry equal timestamps.

Time series data provides a historical context to the analysis typically associated with complex event processing. This can apply to any vertical industry such as finance<ref>[{{cite web|url=http://cs.nyu.edu/shasha/papers/jagtalk.html "|title=Time Series in Finance", Retrieved May 16, 2012]|website=cs.nyu.edu}}</ref> and cooperatively with other technologies such as BPM.

Consider the scenario in finance where there is a need to understand historic price volatility to determine statistical thresholds of future price movements. This is helpful for both trade models and transaction cost analysis.

The ideal case for CEP analysis is to view historical time series and real-time streaming data as a single time continuum. What happened yesterday, last week or last month is simply an extension of what is occurring today and what may occur in the future. An example may involve comparing current market volumes to historic volumes, prices and volatility for trade execution logic. Or the need to act upon live market prices may involve comparisons to benchmarks that include sector and index movements, whose intra-day and historic trends gauge volatility and smooth outliers.

==Internet of Thingsthings and Smartsmart Cybercyber-physical systems==

Complex event processing is a key enabler in [[Internet of Thingsthings]] (IoT) settings and Smartsmart [[Cybercyber-physical system]]s (CPS) as well. Processing dense and heterogeneous streams from various sensors and matching patterns against those streams is a typical task in such cases.<ref>[{{cite web|url=http://msdl.cs.mcgill.ca/people/istvan/pub/mtcps2016 |title=Balogh, Dávid, Ráth, Varró, Vörös: Distributed and Heterogeneous Event-based Monitoring in Smart Cyber-Physical Systems, In 1st Workshop on Monitoring and Testing of Cyber-Physical Systems, Vienna, Austria. 2016.]}}</ref> The majority of these techniques rely on the fact that representing the IoT system's state and its changes is more efficient in the form of a data stream, instead of having a static, materialized model. Reasoning over such stream-based models fundamentally differs from traditional reasoning techniques and typically require the combination of [[model transformation]]s and CEP.<ref>I. Dávid, I. Ráth, D. Varró: Foundations for Streaming Model Transformations by Complex Event Processing, International Journal on Software and Systems Modeling, pp 1--28, 2016. {{doi|10.1007/s10270-016-0533-1}}</ref>

==See also==

Line 94 ⟶ 87:

===Vendors and products===

* ''[[Apama'' (software)|Apama]] by [[Software AG]] - monitors rapidly moving event streams, detects and analyzes important patterns, and takes action according to rules.<ref>[http://www.softwareag.com/corporate/products/bigdata/apama/overview/default.asp Apama Real-Time Analytics Overview] {{Webarchive|url=https://web.archive.org/web/20151025155227/http://www.softwareag.com/corporate/products/bigdata/apama/overview/default.asp |date=2015-10-25 }}. Softwareag.com. Retrieved on 2013-09-18.</ref>

* [[Azure Stream Analytics]]

* [[Drools|Drools Fusion]]

* [[Esper (software)|Esper]] Complex event processing for Java and C# (GPLv2).

* [[Feedzai|Feedzai - Pulse]]

* [[Microsoft|Microsoft StreamInsight]] Microsoft CEP Engine implementation<ref>[{{cite web|url=https://technet.microsoft.com/en-us/library/ee362541(v=sql.111).aspx |title=Microsoft StreamInsight|website=technet.microsoft.com|date=28 July product2016 page]}}</ref>

* [[GigaSpaces]] XAP

* ''Informatica RulePoint'' by [[Informatica]]

* [[Microsoft|Microsoft StreamInsight]] Microsoft CEP Engine implementation<ref>[https://technet.microsoft.com/en-us/library/ee362541(v=sql.111).aspx Microsoft StreamInsight product page]</ref>

* [[openPDC]] — A set of applications for processing streaming time-series data in real-time.

* [[Oracle SOA Suite|Oracle Event Processing]] - for building applications to filter, correlate, and process events in real time.

* [[SAP SE|SAP ESP]] - A low-latency, rapid development and deployment platform that allows processing multiple streams of data in real time<ref>[{{Cite web |url=http://scn.sap.com/community/developer-center/esp |title=SAP ESP - Developers community] |access-date=2014-07-17 |archive-url=https://web.archive.org/web/20150105095944/http://scn.sap.com/community/developer-center/esp |archive-date=2015-01-05 |url-status=dead }}</ref>

* BRMS - A rules management engine by [[Red Hat]] based on [[Drools]]

* [[Sqlstream|SQLstream]] SQLstream’sSQLstream's stream processing platform, s-Server, provides a relational stream computing platform for analyzing large volumes of service, sensor and machine and log file data in real-time.

* [[SAP SE|SAP ESP]] - A low-latency, rapid development and deployment platform that allows processing multiple streams of data in real time<ref>[http://scn.sap.com/community/developer-center/esp SAP ESP - Developers community]</ref>

* [[SAS ESP]] - A platform that is built for speed to analyse (apply SAS' and third-party analytics, including machine learning algorithms) millions of data records in motion (events) with low-latency response time (milliseconds and sub-milliseconds). Deployable at the edge, on premises and to the Cloud. Flexible platform that is built with openness in mind to make Analytics pervasive everywhere.<ref>https://www.sas.com/en_gb/software/data-management/event-stream-processing.html</ref>

* [[Sqlstream|SQLstream]] SQLstream’s stream processing platform, s-Server, provides a relational stream computing platform for analyzing large volumes of service, sensor and machine and log file data in real-time.

* [[TIBCO| TIBCO BusinessEvents & Streambase ]] - CEP platform and High Performance Low Latency Event Stream Processing

* VIATRA-CEP<ref>https://wiki.eclipse.org/VIATRA/CEP</ref> - A model-driven CEP engine, part of the 3rd generation of the [[VIATRA]] [[model transformation]] framework

* [[WebSphere Business Events]]

* [[Apache Flink]] Open-source distributed stream processing framework with a CEP API<ref>{{cite web|url=https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/cep.html|title=Apache Flink 1.2 Documentation: FlinkCEP - Complex event processing for Flink|website=ci.apache.org}}</ref> for Java and Scala.

* [[WSO2|WSO2 Siddhi]] Complex event processing written in Java. Designed as part of a series of middleware components.

* [[Apache Storm]] Free and open source distributed realtime computation system. Storm processes unbounded streams of data in realtime.

* [[Apache Flink]] Open-source distributed stream processing framework with a CEP API<ref>https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/libs/cep.html</ref> for Java and Scala.

==References==

{{reflist}}

==External links==

* [http://www.complexevents.com Complex Event Processing & Real Time Intelligence]

* [http://www.rfidjournal.com/article/view/1196 Separating the Wheat from the Chaff] Article about CEP as applied to RFID, appeared in RFID Journal

* [http://www.computerworld.com.au/index.php/id;918976660;fp;4;fpid;1398720840 Complex event processing: still on the launch pad in Computerworld]

* [http://www.debs.org The ACM conference on distributed event-based systems] - Research articles on innovations in event-based systems

* [http://www.thecepblog.com/what-is-complex-event-processing/ What is Complex Event Processing?] - A Technical Overview of CEP