BI: Becoming Cheaper, Better, Faster, and More Pervasive Each Year
Technical innovations are making BI more pervasive each and every day.
- By Mike Schiff
- March 12, 2013
As a baby boomer, I continue to chuckle when I hear users complain about a query that takes more than a few seconds to respond. I can still remember when a query involved going to a library, searching a paper card catalogue, and finding and loading microfilm or microfiche into a magnifying reader. I also remember when the closest device to a cell phone (let alone a smartphone) was a two-way wrist radio worn by a comic strip character known as Dick Tracy.
I remember that today's cloud computing craze which, in my opinion, evolved from the time-sharing days of the early 1960s and which, in the mid-1990s, was reborn and greatly enhanced under the guise of network computing and application service providers. The "cloud" initially connected users to mainframe computers over dial-up telephone lines at a rate of 110 baud (approximately 10 characters per second) with teletypes serving as an early user I/O device!
Where We are Today
Thanks to a variety of technical innovations -- including multi-core processors, in-memory technology, 64-bit addressing, solid state storage, big data technologies, and declining hardware prices (in 1970 magnetic core memory cost approximately $1 per byte) -- we can truly process data cheaper and faster. For example, data warehouses originally contained summarized data and data mining was performed on data samples extracted into the data mining tool's specialized database. Today it is not uncommon for a data warehouse to contain highly detailed data and for data mining to be executed on complete (with the exception of data set aside for validation of the resultant model) data sets.
Instead of yesterday's trip to a library, or years later to the glass-house data center, today's mobile technology facilitates the delivery of the right information to the right user at the right time. Thanks to smartphones and tablets, queries can now be made from almost anywhere. Although most end users are certainly not BI experts, interactive visualization techniques such as dashboards with drill-down capabilities have replaced the minimally formatted output tables of yesterday and made results much easier to comprehend and analyze. Results can now be easily reformatted and displayed almost instantly through highly customized graphs and charts that once required a graphic artist. Mobile technology has served to make it easier to access and analyze data and has enabled the promise of "pervasive business intelligence."
Queries once required users to have expertise or, prior to SQL standardization, knowledge of a database-specific proprietary query language, and thus users were dependent on BI or IT "middleman" experts to formulate. This is certainly no longer the case. I used to jokingly speculate that BI tools were invented to shield end users from SQL; however, the user-friendly interfaces of today are frequently English-language, touch screen, or even voice-based. Although Apple's voice-response Siri interface is either loved or hated by most users, I think it (she?) is closely analogous to posing a question to a librarian who tries his or her best to understand what is being asked and then utilizes the library's resources to find an answer.
Anticipate Future Technology Improvements
Today, BI products and services are capable of analyzing vast quantities of data at a fraction of the time that it would have taken to analyze much less data only a few years earlier. This will only get faster in the future. I encourage organizations to take advantage of inexpensive storage to collect more detailed data than they would have just a few years ago.
For example, organizations may have originally stored product sales details by customer and/or by contact type (e.g., retail store, website, mail order, telephone). Today they are likely also collecting data relative to user website activity including items removed from shopping carts, website navigation, and website searches, in order to determine what their potential customers looked at but did not purchase. If they are not doing so today, they will likely soon be analyzing related social media data as well.
Given the continually declining price curve on storage, organizations should also collect data and archive data that they might not have immediate use for today; it is likely that some of this data will be useful for analyses that they don't anticipate today. As technology improvements continue to make BI cheaper, better, faster, and more pervasive, future data mining of this data could yield nuggets of gold that might not be discoverable today.
Consider the television series Person of Interest in which the heroes use a computer with a large variety of feeds (i.e., input data) to anticipate and prevent crimes and acts of terror. Also consider how technology such as IBM's Watson is currently analyzing vast quantities of clinical patient data to gain insights into cancer treatment or how technology is being deployed to develop personalized drugs based on an individual's genomic data and perhaps other, yet to be discovered, factors.
Advances in BI will continue, and yesterday's science fiction is likely to become tomorrow's reality.