Why Organizations Must Get the Small Stuff Right
Rather than worrying about new technologies, organizations should be sweating the small stuff: the bread-and-butter activities such as operational reporting or scorecards that still perplex so many enterprises.
- By Stephen Swoyer
- November 12, 2013
Some seem ready to close the book -- or to slam shut the door -- on the data warehouse (DW) and its wide assortment of business intelligence (BI) tools.
With big data, advanced analytics, and the state-less, REST-ful cloud, we've turned to a new chapter in information management. In all three cases, after all, data is generated and consumed in languages or by means of transport mechanisms other than SQL.
TDWI's recent World Conference in San Diego served up anecdotal notice of a counterrevolution, however. To be sure, the conference featured three big data-themed educational tracks, but only one of TDWI's 2013 Best Practices Award Winners (Motorola Mobility) explicitly emphasized the use of a big data technology platform. Most award winners involved variations on data warehouse-oriented projects.
That's precisely the point, according to Michael Whitehead, CEO of data integration specialist WhereScape Inc.: we should be sweating the small stuff.
"We're still not getting the really 'basic' stuff done," said Whitehead, citing operational reporting, scorecards, and other analytic assets. "A lot of [organizations] are still struggling with reporting, with dashboards. After how many years, they don't even have that. Now we're telling them they should move on to this other thing?"
WhereScape doesn't ignore or reject big data, he protests: its RED integration platform connects to and supports drag-and-drop data movement from Hadoop (HBase and Hive) and other NoSQL platforms. However, Whitehead has called big data "a distraction." He expanded on what he meant by this at TDWI's World Conference: "Big data is now a C-level [topic of] discussion. We [as an industry] all need it, because we're all relevant again. We haven't been relevant in 15 years. We started out with a hiss and a roar with decision support. It was all 'You have to get a data warehouse!' and all of that stuff. That has gone down, down, down [in cachet] in the last ten years as we've failed to deliver on most of our promises. Now we have this shiny new thing: 'big data.' The good thing is that [thanks to big data] IT's gone from a cost center to a strategic asset -- but we need to take that momentum and not muck it up," Whitehead said.
WhereScape's prescriptive response to this is automation. Automation addresses what's wrong with the data warehouse -- e.g., its brittleness and inflexibility -- as well as the spate of new challenges served up by big data, Whitehead argued. In both cases, "automation" means eliminating manual, repetitive, or tedious tasks: e.g., WhereScape RED can automatically prepare and move data (e.g., extracting data from source systems, loading it into a target platform, building dimensions, generating aggregates, and creating a semantic layer) as well as generate and manage documentation.
WhereScape likewise proposes to reduce or eliminate the hand-coding -- in Java, Pig Latin, or in other procedural or scripting languages -- that's still de rigueur in the Hadoop world. "The minute that you start contemplating some sort of big data-style solution, you've just added another bit of technology that has its own tools, its own people -- its own [supporting] infrastructure."
Whitehead's isn't the only voice pleading for restraint. Charles Caldwell, director of solutions engineering with Logi Analytics (the former LogiXML) articulated a similar viewpoint.
"We're getting uptake in large enterprises ... where folks have implemented BI tools and tried to [incorporate them] into their applications, and what happens is ... the BI architectures just aren't working for them. They're unusable," said Caldwell, who explained that Logi Analytics focuses on embedding into targeted, user-focused applications.
The Logi Info platform (Logi Analytics' flagship offering) permits application developers to expose BI or analytic capabilities in domain- or function-specific applications, Caldwell indicated. As he sees it, traditional BI tools want to take over and effectively to dictate the business process, he argues; instead of embedding into a process as is, they require that it be changed. This, he said, is the chief reason BI adoption continues chronically to lag. "We come into customers where they've implemented a BI tool, with a traditional data warehouse, and it just isn't doing what they want it to do. It can't do what they want it to do," he said.
It's the difference between specialization and generalization, Caldwell argued: i.e., the difference between a single, targeted or function-specific tool -- much like the apps you'd buy for your smartphone -- or a general-purpose tool chest. "The reason you have 40-45 apps on your iPhone is that they do what you want them to do. They're designed for specific purposes. At the same time, you're ecstatic not to have a generalized multi-purpose app that doesn't do exactly what you want it to do -- that doesn't do any one thing very well at all," he notes.
"The key question is how do I allow you to see the specific needs of a group of your users and to meet those needs in a very targeted way?" Caldwell concluded. "That's the question we as an industry have ignored for far too long. Whether that's on mobile, on the PC, or embedded in another app, we have to be able to accommodate that. Traditionally, the industry has done an extremely poor job with this. That's created an opening for us."
Everybody, said WhereScape's Whitehead, needs to think about what they're doing -- or what they're promising to do -- with big data. Because of the sheer amount of hype, because of the C-level interest, he argued, there's a big opportunity for vendors, consultants, integrators, and IT teams to boost their budgets, prestige, and/or profit margins. There's a big danger, too. "If we aren't careful about this [i.e., big data], we're going to screw it up worse than we screwed up the data warehouse revolution," he concluded.