Choosing the Right Time for Real Time
In the world of real-time BI, faster may not be better. We offer three issues to consider as you pursue real-time BI and analytics.
- By David Stodder
- May 20, 2014
The pursuit of "real time" has long been a driving force in business intelligence, analytics, and data management in part because for much of this technology's history, users' data views have been stuck in the past. Traditional BI reports presented only historical data; new data loads or deeper analytical queries could only be handled during off-hours batch processes. There is considerable pent-up demand for anything closer to real-time data access, analysis, and presentation.
Fortunately, technology advances are enabling organizations to close the data latency gap. BI tools and underlying operational data stores, federated systems, in-memory analytics platforms, and extract, load, and transformation (ELT) solutions can support far more frequent updating. Exciting technologies are reshaping Hadoop environments that have been limited to batch-oriented analytics: Apache Hadoop 2.0, Spark, Shark, YARN, and commercial technologies such as Cloudera Impala.
"Real time" of course, means different things depending on users and their circumstances, so expectations need to be managed carefully. Faster always sounds better, and users, being human, are susceptible to the hype of instant gratification. Although Hadoop-related technologies could offer a less-expensive alternative, costs usually rise as organizations push their technology stacks in the direction of real-time data access, analytics, and presentation. Users also need to be informed that data quality will likely fall below standards because there's usually no time to run data quality, profiling, and cleansing processes.
However, for some objectives, the additional costs and travails of achieving real time are worth it. A more timely view of online behavior, for example, could mean the difference between success and failure in next-best-action marketing. New channels such as social media demand real-time engagement. Organizations need to continuously learn from online behavior so they can adjust to trends and make changes to customer services when events (such as bad weather) cause delays in shipment or fulfillment.
Event streams and other "data in motion" sources can no longer be ignored by organizations in utilities, healthcare, financial services, and other industries. To interdict fraud, manage costs, and allocate resources, many such firms need to perform predictive analytics and tap data streams to establish smarter operational monitoring.
At the recent TDWI Big Data Analytics Solution Summit in Savannah, Georgia, we heard a terrific presentation by Dr. Timothy Buchman, MD, Ph.D., about Emory Critical Care Center's emerging "intensive care unit (ICU) of the future," as he termed it. Buchman detailed how and why Emory is employing stream processing and analytics (using IBM InfoSphere Streams) to improve the Emory ICU's situational awareness about patients' conditions. Real-time information can enable healthcare providers to intervene with patients earlier, when less-complicated care can solve problems before conditions worsen.
Gaining real-time insight from data in motion is part of the never-ending competitive battle among financial services firms to be first to know which events will affect stock prices so they can adjust trading strategies accordingly. Some utilities are experimenting with stream analytics to examine the rush of data generated by smart meters and smart grids. Game developers have long been at the cutting edge of real-time analytics and monitoring; they need to understand gamers' experiences so they can fix faults in current games and develop new ones rapidly.
Let Cool Heads Prevail
To most business executives and technology professionals, faster is simply better. Yet, organizations should resist getting caught up in the hype. It is important to consider issues that may crop up during the pursuit of real-time BI and analytics. Here are three:
#1. Not all the data needs to be real time. BI directors and data warehouse managers need to guide users so that real-time requests are realistic and will deliver business value. Focus attention on areas where decision cycles are fastest.
#2. Evaluate technology for availability and reliability. Before users sink resources into real-time analytics, data managers need assurances that the data flow will be uninterrupted. Work with IT to ensure that underlying systems and networks are prepared to support real-time data demands.
#3. Don't assume that application business rules can handle real time. It's possible that older business rules in applications will need to be rewritten. Make sure developers are able to synchronize rules in applications with real-time data requirements for BI and analytics. This can turn into a "gotcha" that frustrates ambitions for real time.
TDWI Research into Real Time
The real-time trend in BI, analytics, and data management is of great interest to TDWI. In collaboration with my TDWI Research colleagues Philip Russom and Fern Halper, I will be working in the coming weeks on a new TDWI Best Practices Report that will focus on real-time BI, analytics, and data management. Look for news about this report at www.tdwi.org.