Complex Event Processing (CEP) is useful for big data because it is intended to manage data in motion. Complex Event Processing is a technique for tracking, analyzing, and processing data as an event happens. This information is then processed and communicated based on business rules and processes.
The idea behind CEP is to be able to establish the correlation between streams of information and match the resulting pattern with defined behaviors such as mitigating a threat or seizing an opportunity. CEP is an advanced approach based on simple event processing that collects and combines data from different relevant sources to discover events and patterns that can result in action.
Here is an example. A retail chain creates a tiered loyalty program to increase repeat sales — especially for customers who spend more than $1,000 a year. It is important that the company creates a platform that could keep these critical customers coming back. Using a CEP platform, as soon as a high-valued customer uses the program, the system triggers a process that offers the customer an extra discount.
Another process rule could give the customer a surprise — an extra discount or a new product sample. The company also adds a new loyalty program that links to a mobile application. When a loyal customer walks near a store, a text message offers the customer a discounted price. If that loyal customer writes something negative on a social media site, the customer care department is notified and issues an apology.
It is quite likely that you are dealing with a huge number of customers with a significant number of interactions. But it would not be enough to simply stream the data and analyze that data. To achieve the business goals the retailer wanted to achieve would require executing a process to respond to the results of the analysis.
Many industries take advantage of CEP. Credit card companies use CEP to better manage fraud. When a pattern of fraud emerges, the company can shut off the credit card before the company experiences significant losses. The underlying system will correlate the incoming transactions, track the stream of event data, and trigger a process. CEP is also implemented in financial-trading applications, weather-reporting applications, and sales management applications, to name a few.
What all these applications have in common is that the applications have a predefined norm for temperature, pressure, size of transaction, or value of the sale. A change in state will trigger an action. If you drive a late-model car, you probably have noticed that when a tire’s pressure has dropped, the car will trigger a dashboard indicator that notifies the driver to take action (getting the tire fixed).
Many vendors offer CEP solutions. Many of the CEP tools on the market allow the creation of real-time, event-driven applications. These applications might ingest data from streams, but they can also ingest data from traditional database sources. Most of the offerings include common capabilities, including a graphical development environment that is typically Eclipse-based, connectivity to real-time data flows, as well as APIs to historical data sources.
Most of these products include a graphical event flow language and support SQL. Key vendors in this space include Esper (open source vendor), IBM with IBM Operational Decision Manager, Informatica with RulePoint, Oracle with its Complex Event Processing Solution, Microsoft’s StreamInsights, and SAS DataFlux Event Stream Processing Engine, and Streambase’s CEP. Numerous startups are emerging in this market.
